spambayes

http://www.linuxjournal.com/article/6466?page=0,1

Components
Three classifier programs are in the Spambayes software: a procmail filter, a POP3 proxy and a plugin for Microsoft Outlook 2000. I cover the procmail filter and the POP3 proxy in this article. A web interface (covered below) and various command-line utilities, test harnesses and so on are also part of Spambayes; see the documentation that comes with the software for full details.

Procmail-Based Setup
If you use a procmail-based e-mail system, this is how the Spambayes procmail system works:

All your existing mail has a new X-Spambayes-Trained header. The software uses this to keep track of which messages it has already learned about.

The software looks at all your incoming mail. Messages it thinks are spam are put in a “spam” mail folder. Everything else is delivered normally.

Every morning, it goes through your mail folders and trains itself on any new messages. It also picks up mail that’s been refiled—something it thought was ham but was actually spam and vice versa. Be sure to keep spam in your spam folder for at least a day or two before deleting it. We suggest keeping a few hundred messages, in case you need to retrain the software.

You’ll need a working crond to set up the daily training job. Optionally, you can have a mailbox of spam and a mailbox of ham to do some initial training.

To set up Spambayes on your procmail system, begin by installing the software. I’ll assume you’ve put it in $HOME/src/spambayes. Then, create a new database:

$HOME/src/spambayes/hammiefilter.py -n
If you exercise the option to train Spambayes on your existing mail, type:

$HOME/src/spambayes/mboxtrain.py \
-d $HOME/.hammiedb -g $HOME/Mail/inbox \
-s $HOME/Mail/spam
You can add additional folder names if you like, using -g for good mail folders and -s for spam folders. Next, you need to add the following two recipes to the top of your .procmailrc file:
:0fw
| $HOME/src/spambayes/hammiefilter.py
:0

  • ^X-Spambayes-Classification: spam
    $HOME/Maildir/.spam/
    The previous recipe is for the Maildir message format. If you need mbox (the default on many systems) or MH, the second recipe should look something like this:
    :0:
  • ^X-Spambayes-Classification: spam
    $HOME/Mail/spam
    If you’re not sure what format you should use, ask your system administrator. If you are the system administrator, check the documentation of your mail program. Most modern mail programs can handle both Maildir and mbox.
    Using crontab -e, add the following cron job to train Spambayes on new or refiled messages every morning at 2:21 AM:

21 2 * $HOME/src/spambayes/mboxtrain.py -d
$HOME/.hammiedb -g $HOME/Mail/inbox
-s $HOME/Mail/spam
You also can add additional folder names here. It’s important to do this if you regularly file mail in different folders; otherwise Spambayes never learns anything about those messages.

Spambayes should now be filtering all your mail and training itself on your mailboxes. But occasionally a message is misfiled. Simply move that message to the correct folder, and Spambayes learns from its mistake the next morning.

Many thanks to Neale Pickett for the information in this section.

procmailrc2


$ cat .procmailrc

:0 HB
* Alanine|Cysteine|Glycine|Histidine
* ^From.*[email protected]
{
        :0c:
        | (formail -r -A"X-Loop: [email protected]" -I"From: [email protected]"; cat help.txt ) | $SENDMAIL -t
        :0
        | (formail -r -A"X-Loop: [email protected]" -I"From: [email protected]" -I"To: [email protected]" ; cat help.txt ) | $SENDMAIL -t
}


$ cat help.txt 
Hi,

do you need systems help with said meeting/presentation? if so, do let us know what you need..

best regards,
liang zhu

remote access with ssh

teamviewer or ngrok are yet another vulnerability layer and crappy software from fly-by-night developers that run on our computers so lets not use them.

lets stick to openssh-server. a yum or apt-get install away.

from http://blog.trackets.com/2014/05/17/ssh-tunnel-local-and-remote-port-forwarding-explained-with-examples.html

assuming our home pc’s public,external, web facing ip address is 1.2.3.4, we have a user lz on the home pc and a user liangzhu in the bii pc:

on the bii pc, do
ssh -R 9000:localhost:22 [email protected]
then leave it connected and then go home

at home, on your home pc, do
ssh -p 9000 [email protected]

on your bii pc’s /etc/ssh/sshd_conf you might need to add this line

GatewayPorts yes

and do a
service sshd reload
for this to work

slowpc

apt-get install cpufrequtils preload

in /etc/sysctl.conf

vm.swappiness=10
vm.vfs_cache_pressure=100
vm.dirty_expire_centisecs = 1000
vm.dirty_writeback_centisecs = 500
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.overcommit_ratio=99
vm.overcommit_memory=2

in rc.local

By default this script does nothing.

echo noop > /sys/block/$DEVICE/queue/scheduler
echo 2 > /sys/block/sda/queue/rq_affinity
echo 0 > /sys/block/sda/queue/add_random
for each in 0 1; do echo performance > /sys/devices/system/cpu/cpu$each/cpufreq/scaling_governor; done

exit 0

procmailrc


INCLUDERC=/mnt/vacation/liangzhu/rc.vacation

:0 Whc: msgid.lock
| formail -D 8192 msgid.cache

#:0
#* ^(To|CC):[email protected]
#Maildir/enquiry

# copy gmail.
:0c:
* ? formail -x"From" -x"From:" -x"Sender:" \
    -x"Reply-To:" -x"Return-Path:" -x"To:" \
    | egrep -is -f ~/white.lst
[email protected]

:0
* ? formail -x"From" -x"From:" -x"Sender:" \
    -x"Reply-To:" -x"Return-Path:" -x"To:" \
    | egrep -is -f ~/white.lst
Maildir/

:0
* ? formail -x"From" -x"From:" -x"Sender:" \
    -x"Reply-To:" -x"Return-Path:" -x"To:" \
    | egrep -is -f ~/black.lst
Maildir/.Junk/

:0c: B
* liangzhu
[email protected]

:0 B
* liangzhu
Maildir/

:0
* ^Subject: Password expiring in
Maildir/.Password/

#:0 B
#* Please change your BII password within
#Maildir/.Password/

#:0 B
#* If you see negative days,please change your password NOW
#Maildir/.Password/

#:0
#* !^From.*bii.a-star.edu.sg
#* ^From.*a-star.edu.sg
#Maildir/.astar/

# copy gmail.
:0c:
[email protected]

# Spam tagged mail goes into spam folder
:0
* ^X-Spam-Flag: YES*
Maildir/.Spam/


# All other mail goes into the main inbox.
:0
Maildir/

how is amber related?

how is amber related? what on earth is this doing? does amber depend on cuda? how about tensorflow?


#!/usr/bin/env bash

if [ $# = 0 ]; then
    # circleci build
    echo "CIRCLE_BRANCH", $CIRCLE_BRANCH
    buildfull=`python -c "import os;  print(os.getenv('CIRCLE_BRANCH').startswith('circleci_'))"`
    echo 'buildfull = ' $buildfull

    if [ "$buildfull" == "True" ]; then
        AMBER_BUILD_TASK='ambertools'
        pyversion=`python -c "import os; env=os.getenv('CIRCLE_BRANCH'); print(env.strip('circleci_'))"`
        echo "pyversion = " $pyversion
    else
        AMBER_BUILD_TASK='ambermini'
    fi
else
    # e.g: build ambertools with python=2.7
    # bash scripts/run_docker_build.sh ambertools 2.7
    AMBER_BUILD_TASK=$1 # ambertools or ambermini
    pyversion=$2
fi

echo "AMBER_BUILD_TASK = " $AMBER_BUILD_TASK
echo "Python version = " $pyversion

FEEDSTOCK_ROOT=$(cd "$(dirname "$0")/.."; pwd;)
echo "FEEDSTOCK_ROOT" $FEEDSTOCK_ROOT
DOCKER_IMAGE=ambermd/amber-build-box
BZ2FILE=/root/miniconda3/conda-bld/linux-64/amber*.tar.bz2

docker info

cat << EOF | docker run -i \
                        --rm \
                        -v ${FEEDSTOCK_ROOT}:/feedstock_root \
                        -a stdin -a stdout -a stderr \
                        $DOCKER_IMAGE \
                        bash || exit $?
    export PATH=/root/miniconda3/bin:\$PATH
    conda update --all --yes
    export AMBER_BUILD_TASK=${AMBER_BUILD_TASK}
    echo "Building" \${AMBER_BUILD_TASK}
    export SKIP_REGISTRATION=True
    if [ "\${AMBER_BUILD_TASK}" == 'ambermini' ]; then
        # build in a single containter
        conda build /feedstock_root/recipe --py 2.7 --quiet || exit 1
        conda build /feedstock_root/recipe --py 3.4 --quiet || exit 1
        conda build /feedstock_root/recipe --py 3.5 --quiet || exit 1
    else
        # build whole ambertools, for a single python version
        # should build each ambertools python version on each branch
        conda build /feedstock_root/recipe --py $pyversion --quiet || exit 1
    fi
    cp $BZ2FILE /feedstock_root/
    echo "done. Please check amber*tar.bz2 files in " $FEEDSTOCK_ROOT
EOF

docker2

~# nvidia-docker run -it -p 8888:8888 tensorflow/tensorflow:1.1.0-devel-gpu-py3
1.1.0-devel-gpu-py3: Pulling from tensorflow/tensorflow
c62795f78da9: Pull complete
d4fceeeb758e: Pull complete
5c9125a401ae: Pull complete
0062f774e994: Pull complete
6b33fd031fac: Pull complete
7ab5dd833cf2: Pull complete
df9cc763fcde: Pull complete
9b0174a3640e: Pull complete
1efd10acdd72: Pull complete
f77b671e3092: Pull complete
af9093817c44: Pull complete
b2fb381211f0: Pull complete
af9e216e9ceb: Pull complete
09860f91cae6: Pull complete
9f0f8d45045d: Pull complete
0467b7b5b3c8: Pull complete
311d6a376dba: Pull complete
d775db884640: Pull complete
befeab6b5e5d: Pull complete
7dd802ca32d9: Pull complete
d1a48577f024: Pull complete
9e3601307c1a: Pull complete
057615e69ff0: Pull complete
Digest: sha256:8eaa148ede85f6689aa3ee5c6d6d34e5b05fdc9af7c293a56f59887aedb851d2
Status: Downloaded newer image for tensorflow/tensorflow:1.1.0-devel-gpu-py3
nvidia-docker | 2017/10/17 10:32:09 Error: unsupported CUDA version: driver 6.5 < image 8.0.61

docker

when i do

~$ docker build –pull -t $USER/tensorflow-serving-devel -f Dockerfile.devel .

it doesn’t finish…stole Dockerfile.devel fromhttps://github.com/tensorflow/serving/blob/master/tensorflow_serving/tools/docker/Dockerfile.devel



FROM ubuntu:16.04

MAINTAINER Jeremiah Harmsen <[email protected]>

RUN apt-get update && apt-get install -y \
        build-essential \
        curl \
        git \
        libfreetype6-dev \
        libpng12-dev \
        libzmq3-dev \
        mlocate \
        pkg-config \
        python-dev \
        python-numpy \
        python-pip \
        software-properties-common \
        swig \
        zip \
        zlib1g-dev \
        libcurl3-dev \
        openjdk-8-jdk\
        openjdk-8-jre-headless \
        wget \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Set up grpc

RUN pip install mock grpcio

# Set up Bazel.

ENV BAZELRC /root/.bazelrc
# Install the most recent bazel release.
ENV BAZEL_VERSION 0.5.1
WORKDIR /
RUN mkdir /bazel && \
    cd /bazel && \
    curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && \
    chmod +x bazel-*.sh && \
    ./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    cd / && \
    rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh

CMD ["/bin/bash"]

</[email protected]>

but its pulled…

# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nvidia/cuda latest 9d3688f974a1 4 days ago 1.73GB
hello-world latest 05a3bd381fc2 4 weeks ago 1.84kB
tensorflow/tensorflow latest-gpu 4ca1d30fcd9b 2 months ago 3.28GB
nvidia/cuda 6.5 794ce05b823b 10 months ago 1.02GB

because i using cuda 6.5 on my “host” the latest one doesnt work, have to get the one tagged 6.5

# nvidia-docker run –rm -ti nvidia/cuda:6.5 nvcc –version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Wed_Aug_27_10:36:36_CDT_2014
Cuda compilation tools, release 6.5, V6.5.16

so what next?

cudnn90

shucks this is the max cuda for your nvidia driver version..
at least 361.00 is required for CUDA 8.0
at least 352.00 is required for CUDA 7.5
at least 346.00 is required for CUDA 7.0
at least 340.00 is required for CUDA 6.5
at least 331.00 is required for CUDA 6.0
at least 319.00 is required for CUDA 5.5

  • apt-get install ansible

and then

  • maybe save the below as cuda.yml and run ansible-playbook cuda.yml ?

- hosts: localhost
  #user: somebody
  #sudo: yes
  tasks:
    - name: register CUDA repository
      shell: wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64-deb && dpkg -i local_installers/cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64.deb

    - name: install CUDA
      apt: name=cuda update_cache=yes

    - name: write environmant variables to bash profile
      copy: src=./cuda.sh dest=/etc/profile.d/

    - pip:
        name: numpy

    - pip:
        name: Keras

    - pip:
        name: tensorflow-gpu

    #- pip:
    #    name: --upgrade $TF_BINARY_URL
    #  environment:
    #    CUDA_HOME: /cluster/apps/x86_64/packages/cuda_8.0
    #    TF_BINARY_URL: https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0rc1-cp27-none-linux_x86_64.whl

  • install cudnn

cudnn-9.0-linux-x64-v7

libcudnn7_7.0.2.43-1+cuda9.0_amd64

libcudnn7-dev_7.0.2.43-1+cuda9.0_amd64

cudnn-6.5-linux-x64-v2

cudnn-sample-v2