Setting up virtualenvwrapper with pyenv using pyenv-virtualenvwrapper
Typically on OS X, I’m prone to use pyenv to manage multiple Python versions, particularly 2.7 and 3.3. I always keep the 2.7.x Python (system version) that ships with OS X, and install extra versions using pyenv, and develop under virtualenvs. It seems virtualenv and virtualenvwrapper do not natively work well with pyenv’s shims when creating new virtualenvs, and it also seems the author of pyenv already gives the solution, pyenv-virtualenvwrapper.
As an essential record, install Python 3.4.1 using pyenv (for installation of pyenv itself, see its documentation) and enable it globally, along with the system version.
$ pyenv install 3.4.1
$ pyenv rehash
$ pyenv global system 3.4.1
You can verify current available versions using the following command:
$ pyenv versions
* system (set by /Users/shichao/.python-version)
* 3.4.1 (set by /Users/shichao/.python-version)
Make sure you have installed virtualenvwrapper and pyenv-virtualenvwrapper (using brew, of course):
$ pip install virtualenvwrapper
$ brew install pyenv-virtualenvwrapper
Then, put the following lines somewhere in your ~/.bash_profile:
# pyenv
eval "$(pyenv init -)"
# virtualenvwrapper
export WORKON_HOME=~/envs
Then you can enable virtualenvwrapper with pyenv virtualenvwrapper, and you may set an alias for that.
Now, you can normally create new virtualenvs.
Create a virtualenv from the system version:
$ mkvirtualenv env1 -p $(which python)
Similarly, create a virtualenv from 3.4:
$ mkvirtualenv env2 -p $(which python3.4)
Fixing MongoDB against unclean shutdown
It’s been a while since I maintained one of my servers that host an old project. After I got an HTTP 500 in an occasional case, I logged in to check what happened, only to find MongoDB is down. Running service mongod restart does not work. The MongoDB log indicates “Unclean shutdown detected.”
After a search on Google, here’s what I did to fix that:
# grep dbpath /etc/mongod.conf
dbpath=/var/lib/mongo
# rm /var/lib/mongo/mongod.lock
# mongod --repair --dbpath /var/lib/mongo
At this point, you will see that MongoDB is running the repair process. If you immediately start MongoDB server after that, it still fails, because some permissions in /var/lib/mongo have been changed. You have to recover owner and group of those files back to “mongod” and then start the service:
# chown -R mongod:mongod /var/lib/mongo
# service mongod restart
Finally note that this is an CentOS server. If you use MongoDB on Ubuntu, some namings may be different.
Get number of bits of a Python integer
A Python integer is represented as either of 32-bit or 64-bit signed integer type that is platform-specific, without considering it as an Python object. As indicated in the documentation, whether it is 32-bit or 64-bit can be revealed by checking sys.maxsize (or sys.maxint on older CPython implementations such as 2.4), the value of which “is the largest positive integer supported by the platform’s Py_ssize_t type and is usually 2**31 - 1 on a 32-bit platform and 2**63 - 1 on a 64-bit platform”. [1] [2].
The following function helps determine number of bits of an integer and returns either 64 or 32 by leveraging the variable discussed above:
import sys
def get_int_bits():
if hasattr(sys, 'maxsize'):
max_int = getattr(sys, 'maxsize')
else:
max_int = getattr(sys, 'maxint')
if max_int >> 62 == 0:
return 32
else:
return 64
Check out the code on GitHub Gist from this link.
UPDATE: June 10, 2014
After an exploration of the built-in types, it is clear that int.bit_length can be used more easily than bit manipulation.:
import sys
def get_int_bits():
if hasattr(sys, 'maxsize'):
max_int = getattr(sys, 'maxsize')
else:
max_int = getattr(sys, 'maxint')
return max_int.bit_length() + 1
Using Bash arrays and IFS to manipulate filenames
Assume we want to work with files in a directory. The most intuitive way is to make use of filename expansion, using globbing. We can use the for construct and wildcard * to handle each file nice and clean, without worrying about those filenames that contain whitespaces (suppose that in all of the following context, the working directory contain three files: “Foo”, “Bar”, and “Foo Bar”):
$ for file in *; do echo "$file"; done
We can also create an array from these filenames, using the following command:
$ files=(*)
Then, we can take a look at both the attributes and the value of files:
$ declare -p files
declare -a files='([0]="Bar" [1]="Foo" [2]="Foo Bar")'
What if we want to create an array with specific orders, say by modification time (mtime) of files. We have to use the ls with the -t switch. Now there are several practical scenarios discussed as follows.
Get the latest modified filename:
$ filename=$(ls -t | head -1)Create an array of filenames ordered by their mtime. Clearly, globbing will not work because it is based on filename pattern matching. We have to use both command substitution and word splitting. Let’s try the following first:
$ filenames=($(ls -t)) $ declare -p filenames declare -a filenames='([0]="Foo" [1]="Bar" [2]="Bar" [3]="Foo")'
Clearly this is not the result we want since the whitespace-contained filename is corrupted.
We need to hack a little on IFS variable. Let’s try what happens if we disable word splitting by setting IFS to null:
$ IFS= $ filenames=($(ls -t)) $ declare -p filenames declare -a filenames='([0]="Foo Bar Bar Foo")'
It seems it’s getting worse this time. Since word splitting is not performed, the result of command substitution becomes an single word.
We know that the result of ls command is multiple lines of filenames. As the result of command substitution (i.e. $(ls -t)), the filenames are separated by newlines (\n). The default IFS is a sequence of \t\n (space, tab and newline), which means each of these characters will be used to delimit words. Here we need to suppress the effect of space; just set IFS to newline character (\n) (as long as your filenames does not contain newlines), with ANSI C quoting ($''):
$ IFS=$'\n' $ filenames=($(ls -t)) $ declare -p filenames declare -a filenames='([0]="Foo Bar" [1]="Bar" [2]="Foo")'
In similar ways, we can get n most recently modified filenames, just make use of parameter expansion:
$ echo "${filenames[@]:0:2}" # Print two most recently modified names
Bash double brackets, test contructs and word splitting
We know that the double brackets constructs can be used in place of the test builtin ([) commonly used as single brackets to evaluate conditional expressions. The double brackets are preferred in terms of writing less error-prone scripts, though it is not POSIX-compliant (not portable to many other shells).
The primary difference of double brackets from the test builtin is that word splitting and filename expansion are not performed within them. It is easier to understand the filename expansion difference in this case, let’s take a look at the difference of word splitting.
First, the following command fails because in test constructs the word splitting is performed after parameter expansion (and variable expansion, if any):
$ bash -xc 'foobar="hello world"; [ $foobar = "hello world" ] && echo "yes"' + '[' hello world = 'hello world' ']' bash: line 0: [: too many arguments
In the double brackes, the error will not occur because word splitting is not performed, like the following:
$ bash -xc 'foobar="hello world"; [[ $foobar = "hello world" ]] && echo "yes"' + foobar='hello world' + [[ hello world = \h\e\l\l\o\ \w\o\r\l\d ]] + echo yes yes
We can supress word splitting in the test construct by setting IFS to null, because when IFS is null, word splitting never occurs. This time, this command will succeed:
$ bash -xc 'IFS=""; foobar="hello world"; [ $foobar = "hello world" ] && echo "yes"' + IFS= + foobar='hello world' + '[' 'hello world' = 'hello world' ']' + echo yes yes
Another feature to note is that double brackets treat unset variables and null variable (variable whose value is a null string, i.e. ‘’) differently from the test construct. For example:
$ bash -xc 'IFS=""; foobar=""; [ $foobar = "hello world" ] && echo "yes"' + IFS= + foobar= + '[' = 'hello world' ']' bash: line 0: [: =: unary operator expected $ bash -xc 'IFS=""; foobar=""; [[ $foobar = "hello world" ]] && echo "yes"' + IFS= + foobar= + [[ '' = \h\e\l\l\o\ \w\o\r\l\d ]]
The unset or null variable is converted to null argument ('') as a result in double brackets, while in test constructs it is removed.
You may notice in the second step above that the xtrace output of the conditional expression to the right of the operator is \h\e\l\l\o\ \w\o\r\l\d. This is because when using operators ==, != and the POSIX version =, the string to the right is considered a pattern and pattern matching will be performed. The backslash indicates the each character is literal instead of pattern, because “hello world” is quoted. The following example makes the right string as pattern:
$ bash -xc 'IFS=""; foobar="hello world"; [[ $foobar = hello* ]] && echo "yes"' + IFS= + foobar='hello world' + [[ hello world = hello* ]] + echo yes yes
Enabling Adobe Flash Player plugin for Google Chrome on Fedora
The Google Chrome browser I use was installed from the third-party repository by Google. The default Pepper Flash Player plugin (version 12.0) does not support some websites (such as www.bilibili.tv). I have to install the Adobe Flash Player plugin and enable it for Google Chrome.
1. Install the repository from Adobe (assuming 64-bit):
# yum install http://linuxdownload.adobe.com/adobe-release/adobe-release-x86_64-1.0-1.noarch.rpm
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux
2. Install the flash-plugin package:
# yum install flash-plugin
3. Create symbolic link at Google Chrome’s default directory:
# ln -s /usr/lib64/mozilla/plugins/libflashplayer.so /opt/google/chrome/libflashplayer.so
- Restart Google Chrome, go to “about:plugins” page, toggle “Details” switch at upper right corner, disable PepperFlash plugin and enable the Adobe Flash Player plugin with location “/usr/lib64/flash-plugin/libflashplayer.so”
setuid() and uid, euid and suid
We know that setuid bit allows a program to change its effective uid (euid) upon execution. I want to take a look at how setuid() system call works through the following experiment.
Open a terminal session and run the passwd command (a known setuid executable) in the background (it will be suspended with a “[stopped]” output because it requires password from stdin) and get its PID (let’s assume it’s 10401) with echo $!:
$ passwd &
$ echo $! # 10401
Then, run gdb to attach to this process:
$ sudo gdb -p $!
Open another terminal, the run the following command to get the line that represents the current real UID (uid), effective UID (euid), saved UID (suid) and file system UID (fsuid) of the previous passwd process:
$ watch -n 1 grep -i 'uid' /proc/10401/task/10401/status
Uid: 1000 0 0 0
In the first terminal, you can call seteuid(1000) to change its euid to 1000:
(gdb) call seteuid(1000)
$1 = 0
Then the output of the watch command will become:
Uid: 1000 1000 0 1000
As you see the euid is 1000 and the process becomes unprivileged. With the following command, you will notice that the USER field has been changed to the regular user instead of root:
$ ps -u -p 10401
Suppressing Fedora NetworkManager’s prompt to connect to WPA Enterprise wireless networks
This seems to be an unsolved bug (#982429) still under discussion. Many give temporary solutions that modify the corresponding ifcfg-* file and specify the password, possibly in plaintext form. Some suggest encrypting the password using wpa_passphrase ESSID command. The problematic network here is “nyu” of the New York University. Every time I turned on my laptop on the campus, it prompts with a dialog (though the password is actually stored) for me to click the “Connect” button, which is annoying. However, the encryption method does not work as far as I am concerned with Fedora 20. So I just followed the plaintext way to produce a temporary fix for that::
# cd /etc/sysconfig/network-scripts
# vim ifcfg-nyu
Comment or remove the following line:
IEEE_8021X_PASSWORD_FLAGS=user
Create a new file named keys-nyu in this directory with your password:
IEEE_8021X_PASSWORD='your_password'
Don’t forget to make it read-writable by root only:
# chmod 600 keys-nyu
Then restart your network interface associated with “nyu”::
# ifdown nyu
# ifup nyu
Then, the wireless network should work as expected. Try to logout or reboot to see if it surely works.
Use Artistic Style with vim
Artistic Style is a “Free, Fast and Small Automatic Formatter for C, C++, C++/CLI, C#, and Java Source Code”. It is basically a utility that reformats source code in a conventionally-defined indent and coding style.
First, install the program.
On Ubuntu/Debian:
$ sudo apt-get install astyle
On OS X (with Homebrew):
$ brew install astyle
In order to use the command in a simple manner, you can set options in the options file ~/.astylerc. I use the following options that uses Kernighan & Ritchie (K&R) style. For more supported styles and options, see http://astyle.sourceforge.net/astyle.html.
--style=kr
--indent=spaces=4
--indent-preprocessor
--pad-oper
--pad-header
--max-instatement-indent=40
--align-pointer=name
--align-reference=name
--keep-one-line-statements
--convert-tabs
--max-code-length=79
--pad-method-colon=none
Open vim, gVim or MacVim, and after editing some C/C++ files, use the following command to reformat the source code:
:%!astyle
Synchronize sleep and wakeup of machines on Ethernet LAN from Mac using SleepWatcher
The previous post discusses remote suspend (sleep) and wakeup of machines on Ethernet LAN using pm-utils, GNU Screen and Wake-on-LAN (LAN). This post will discuss the synchronization of such behaviors of local and remote machines.
Suppose you are regularly working at home on a Mac (either MacBook or Mac mini or something else), and you have a spare PC running as an alternative to NAS or workstation on the same Ethernet LAN. If the spare PC is Linux, you may have set up netatalk on it to enable AFP file sharing or use it as a Time Capsule. However, since it is probably a netbook and may work only when your Mac is up, you don’t want it to run all days without doing anything. You can thus synchronize sleep and wakeup of your Mac with that PC.
You can download and install SleepWatcher to monitor sleep and wakeup on Mac and write some scripts upon these events.
Download SleepWatcher (version 2.2 as of writing)
Install it following the ReadMe.rtf instructions
After the SleepWatcher LaunchAgent or LaunchDaemon is loaded, run the following commands to add sleep and wakeup scripts:
$ echo "/path/to/your/sleep_script.sh" > ~/.sleep $ echo "sleep 5; /path/to/your/wakeup_script.sh" > ~/.wakeup $ chmod +x ~/.sleep ~/.wakeup
/path/to/your/sleep_script.sh and /path/to/your/wakeup_script.sh are scripts to suspend and wakeup the remote machine (reference the previous post). sleep 5 performs a delay to make sure the network is up to avoid “send : No route to host” error. 4. Test the setup by sleeping and waking up your Mac and check whether the remote machine does the same