Here is a simple bash script (hereby referred to as “Propagate.sh”) which syncs /var/www/html/ to all of your slave instances. It uses the “aws” command line interface provided by Amazon, which comes default with the Amazon Linux starter AMI.
#The first command line of the script contains the master’s IP, so it does not sync with itself. export LocalIP=Your_Master_IP_Here;
#Get the IPs of all slave instances export NewIPs=`aws ec2 describe-instances | grep '"PrivateIpAddress"' | perl -i -pe 's/(^.*?: "|",?\s*?$)//gm' | sort -u | grep -v $LocalIP`
#Loop over all slave instances for i in $NewIPs; do echo "Syncing to: $i"; #Run an rsync from the master to the slave rsync -aP -e 'ssh -o StrictHostKeyChecking=no' /var/www/html/ root@$i:/var/www/html/; done
You may also want to add “-o UserKnownHostsFile=/dev/null” to the SSH command (directly after “-o StrictHostKeyChecking=no”), as a second EC2 instance may end up having the same IP as a previously terminated instance. Another solution to that problem is syncing the “/etc/ssh/ssh_host_rsa_key*” from the master when an instance initializes, so all instances keep the same SSH fingerprint.
To let other people manually execute this script, you can create a PHP file with the following in it. (Change /var/www/ in all below examples to where you place your Propagate.sh)
If your Propagate.sh needs to be ran as root, which it may if your PHP environment is not run as the user root (usually “apache”), then you need to make sure it CAN run as root without intervention. To do this, add the following to the /etc/sudoers file
Change the user from “apache” to the user which PHP runs as (when running through apache). I included “whoami” as a valid sudoer application for testing purposes. Also, in the sudoers file, if “Defaults requiretty” is turned on, you will need to comment it/turn it off.
While I did not mention it in yesterday's post, I thought I should at least mention it here. There are other ways to keep file systems in sync with each other. This is just a good use case for when you want to keep all instances as separate independent entities. Another solution to many of the previously mentioned problems is using Amazon's new EFS, which is currently still in preview mode.
I was recently hired to move a client's site from our primary server in Houston to the Amazon cloud, as it was about to take a big hit in traffic. The normal setup for this kind of job is pretty straightforward. Move the database over to RDS, set up an AMI of an EC2 instance, a load balancer, and ec2 auto scaling. However, there were a couple of problems I needed to solve this time around for the instances launched via the auto scalar that I had not really needed to do before. This includes syncing the SSH settings and current codebase from the primary instance, as opposed to recreating AMIs every time there was a change. So, long story short, here are the problems and solutions that need to be added before the AMI image is created.
This all assumes you are running as root. Most of these commands should work on any Linux distribution that Amazon has default AMIs for, but some of these may only work in the Amazon and CentOS AMIs.
Pre-setup:
Your first instance that you are creating the AMI from should be a permanent instance. This is important for 2 reasons.
When changing configurations for the auto scalar, if and when your instances are terminated and recreated, this instance will always be available on the load balancer, so there is no downtime.
This instance can act as a central repository for other instances to sync from.
So make sure this instance has an elastic IP assigned to it. From here on out, we will refer to this instance as PrimaryInstance (you can set this physically in the host file, or change it in all scripts to however you want to refer to your elastic IP [most likely through a DNS domain]).
Create your ssh private key for the instances: (For all prompts, use default settings)
Make sure your current ssh authorized_keys contains your new ssh private key:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Make sure your ssh known_hosts includes your primary instance, so all future ssh calls to it are automatically accept it as a known host:
ssh PrimaryInstance -o StrictHostKeyChecking=no
You do not have to finish the login process. This just makes sure our primary instance will be recognized by other instances.
Turn on PermitRootLogin in /etc/ssh/sshd_config and reload the sshd config service sshd reload I just recommend this because it makes life way, way easier. The scripts below assume that you did this.
Create a custom init file that runs on boot to take care of all the commands that need to be run.
#Create the script and make sure the full path (+all other root environment variables) are set when it is ran echo '#!/bin/bash -l' > /etc/rc.d/init.d/custom_init
#Set the script as executable chmod +x /etc/rc.d/init.d/custom_init
#Executes it as one of the last scripts on run level 3 (Multi-user mode with networking) ln -s ../init.d/custom_init /etc/rc.d/rc3.d/S99custom_init
All of the below commands in this post will go into this script.
Allow login via password authentication:
perl -i -pe 's/^PasswordAuthentication.*$/PasswordAuthentication yes/mg' /etc/ssh/sshd_config service sshd reload
Notes: You may not want to do this. It was just required by my client in this case. This is required in the startup script because Amazon likes to mess with the sshd_config (and authorized_keys) in new instances it boots.
Sync SSH settings from the PrimaryInstance:
#Remove the known_hosts file, in case something on the PrimaryInstance has changed that would block ssh commands. rm -f ~/.ssh/known_hosts
#Sync the SSH settings from the PrimaryInstance rsync -e 'ssh -o StrictHostKeyChecking=no' -a root@PrimaryInstance:~/.ssh/ ~/.ssh/
Sync required files from the PrimaryInstance. In this case, the default web root folder:
That's it for the things that need to be configured/added to the instance. From there, create your AMI and launch config, and create/modify your launch group and load balancer.
Also, as a very important note about your load balancer, make sure if you are mirroring its IP on another domain to use a CNAME record, and not the IP in an A record, as the load balancer IP is subject to change.
After a little over a year of waiting, Let’s Encrypt has finally opened its doors to the public! Let’s Encrypt is a free https certificate authority, with the goal of getting the entire web off of http (unencrypted) and on to https. I consider this a very important undertaking, as encryption is one of the best ways we can fight illegal government surveillance. The more out there that is encrypted, the harder it will be to spy on people.
I went ahead and got it up and running on 2 servers today, which was a bit of a pain in the butt. It [no longer] supports Python 2.6, and was also very unhappy with my CentOS 6.4 cPanel install. Also, when you first run the letsencrypt-auto executable script as instructed by the site, it opens up your package manager and immediately starts downloading LOTS of packages. I found this to be quite anti-social, especially as I had not yet seen anywhere, or been warned, that it would do this before I started the install, but oh well. It is convenient. The problem in cPanel was that a specific library, libffi, was causing problems during the install.
To fix the Python problem for all of my servers, I had to install Python 2.7 as an alt Python install so it wouldn’t mess with any existing infrastructure using Python 2.6. After that, I also set the current alias of “python” to “python2.7” so the local shell would pick up on the correct version of Python.
The cPanel lib problem was caused by libffi already being installed as 3.0.9-1.el5.rf, but yum wanted to install its devel package as version 3.0.5-3.2.el6.x86_64 (an older version). It did not like running conflicting versions. All that was needed to fix the problem was to manually download and install the same devel version as the current live version.
Unfortunately, the apache plugin was also not working, so I had to do a manual install with “certonly” and “--webroot”.
And that was it; letsencrypt was ready to go and start signing my domains! You can check out my current certificate, issued today, that currently has 13 domains tied to it!
After version 4.2.0.0 (2014-05-08) of phpMyAdmin, it stopped including table’s keys inline within the create table statement, and instead opted to add all the table keys at the very end of the export file by modifying the tables. (See "rfe #1004 Create indexes at the end in SQL export). This behavior has been annoying to many people, including myself, but I never noticed anyone mentioning a fix. I looked into the source and there is a very simple way to restore this behavior to what it originally was.
Edit the file “phpMyAdmin/libraries/plugins/export/ExportSql.class.php”. In it, the code block starting with the below line needs to be skipped
if (preg_match('@CONSTRAINT|KEY@', $create_query)) {
The easiest way to do this is changing that line to
if (false && preg_match('@CONSTRAINT|KEY@', $create_query)) {
In lieu of using my own custom C++ background services to take care of hot key tasks in Windows, I started using AutoHotKey a while back. While it’s not perfect, and it is missing a lot of Win32 API functionality, I am still able to mostly accomplish what I want in it. I was thinking I should add some of the simple scripts I use here.
Center a string within padding characters and output as key-strokes
Example:
PadText = ~*
Length = 43
Text = Example Text
Result = ~*~*~*~*~*~*~*~*Example Text~*~*~*~*~*~*~*~
;Get the last values IniPath=%A_ScriptDir%\AutoHotKey.ini IniRead,PadText,%IniPath%,CenterString,PadText,- IniRead,NewLength,%IniPath%,CenterString,NewLength,10 IniRead,TheString,%IniPath%,CenterString,TheString,The String
;Get the input InputBox,PadText,Center String,Pad Character,,,,,,,,%PadText% InputBox,NewLength,Center String,New Length,,,,,,,,%NewLength% InputBox,TheString,Center String,String To Center,,,,,,,,%TheString%
;Cancel on blank pad or invalid number if StrLen(PadText)==0 { MsgBox,Pad text cannot be blank return } if NewLength is not integer { MsgBox,New length must be an integer return }
;Save the last values IniWrite,%PadText%,%IniPath%,CenterString,PadText IniWrite,%NewLength%,%IniPath%,CenterString,NewLength IniWrite,%TheString%,%IniPath%,CenterString,TheString
;Truncate initial padding to at least half NewString:=Substr(NewString, 1, Ceil(PadLen/2))
;Add the string NewString.=TheString
;Final padding Loop { if StrLen(NewString)>=NewLength break NewString.=PadText }
;Truncate to proper length NewString:=Substr(NewString, 1, NewLength)
;Output to console Sleep,100 Send %NewString% return
Format rich clipboard text to plain text
clipboard = %clipboard% return
Force window to borderless full screen
Description: This takes the active window, removes all window dressing (titlebar, borders, etc), sets its resolution as 1920x1080, and positions the window at 0x0. In other words, this makes your current window take up the entirety of your primary monitor (assuming it has a resolution of 1920x1080).
Description: Saves the currently active window (by its title) and focused control object within the window; asks the user for a keypress interval and the key to press; starts to continually press the requested key at the requested interval in the original control (or top level window if an active control is not found); stops via the F11 key. Note: I had created this to help me get through the LISA intro multiple times.
;Get the current window and control WinGetActiveTitle, TheTitle ControlGetFocus FocusedControl, %TheTitle% if(ErrorLevel) FocusedControl=ahk_parent
;Get the pause interval InputBox,IntervalTime,Starting script with window '%TheTitle%',Enter pause interval in milliseconds. After submitted`, hold down the key to repeat,,,,,,,,200 if(ErrorLevel || IntervalTime=="") ;Cancel action if blank or cancelled return IntervalTime := IntervalTime+0
;Get the key to keep pressing - Unfortunately, there is no other way I can find to get the currently pressed keycode besides polling all 255 of them Sleep 500 ;Barrier to make sure one of the initialization keys is not grabbed Loop { TestKey := 0 Loop { SetFormat, INTEGER, H HexTextKey := TestKey SetFormat, INTEGER, D VirtKey = % "vk" . SubStr(HexTextKey, 3) if(GetKeyState(VirtKey)=1 || TestKey>255) break TestKey:=TestKey+1 } if(TestKey<=255) break Sleep 500 } VirtKey := GetKeyName(VirtKey)
;If a direction key, remap to the actual key if(TestKey>=0x25 && TestKey<=0x28) VirtKey := SubStr(VirtKey, 7)
;Let the user know their key MsgBox Received key: '%VirtKey%'. You may now let go of the key. Hold F11 to stop the script.
;Continually send the key at the requested interval KeyDelay:=10 SetKeyDelay %KeyDelay% #Interval between up/down keys IntervalTime-=%KeyDelay% Loop { ;Press the key ControlSend, %FocusedControl%, {%VirtKey% Up}{%VirtKey% Down}, %TheTitle%
;Check for the cancel key if(GetKeyState("F11")) break
;Wait the requested interval to press the key again Sleep, %IntervalTime% }
;Let the user know the script has ended MsgBox Ending script with window '%TheTitle%' return
I recently bought the game LISAon Steam, and the humor approach is fascinating. Unfortunately, this approach involves being incredible vague, or outright obtuse, at telling you what is going on, or what is going to happen if you do something. The very first choice you have in the game is whether to choose “Pain” mode or “Normal” mode. It doesn’t tell you anything beyond that. Unfortunately, I interpreted this as “Normal” and “Easy”, and so I chose the former “Pain” mode. One of the “features” of pain mode is that you can only use save points once, and there are only 36 of them in the game, spread very far apart. After I was a few hours into the game, and I realized how much of a bother this was going to be, especially because it meant I had to play in possible multi-hour chunks, not knowing when I would get to stop. I didn’t feel like replaying up until that point, so I decided to do some save game file hacking, as that is part of the fun for me.
DO NOTE, this method involves deleting some of the data in the game file, specifically a bunch of boolean flags, which might cause some events in the save to be “forgotten”, so they will reoccur. At the point of the game I was at, the few deleted flag actions that I encountered didn’t affect anything big or of importance. One example of this is the long-winded character repeats his final soliloquy when you enter his map.
So, to switch from “Pain” mode to “Normal” mode in the save file, do the following:
Your save files are located at %STEAM_FOLDER%/steamapps/common/LISA/Save##.rvdata2
Backup the specific save file you want to edit, just in case.
Open that save file in a hex editor. You might need to be in steam offline mode for the edit to stick.
Search for “@data[”. Immediately following it are the hex character “02 02 02”. Delete them and in their place, add the hex character 0x73 (“s”).
Following the “s” character that you just added are 514 bytes that are either “0”, “T”, or “F”, and then a colon (“:”)
Keep the first 110 of these bytes, and then delete everything up to the colon (which should be 404 bytes).
In the course of my Linux administrative duties (on a cPanel server), I have created multiple scripts to help us out with Exim, our mail transfer agent. These are mostly used to help us fight spam, and determine who is spamming when it occurs.
This monitors the number of emails in the queue, and sends ours admins an email when a limit (1000) is reached. It would need to be run on a schedule (via cron).
#!/bin/bash export AdminEmailList="ADMIN EMAIL LIST SEPARATED BY COMMAS HERE" export Num=`/usr/sbin/exim -bpc` if [ $Num-gt 1000 ];then echo"Too many emails! $Num"| /usr/sbin/sendmail -v "$AdminEmailList" #Here might be a good place to delete emails with “undeliverable” strings within them #Example (See the 3rd script): exim-delete-messages-with 'A message that you sent could not be delivered' fi
This deletes any emails in the queue from or to a specified email address (first parameter). If the address is the recipient, the from must be "<>" (root)
This deletes any emails in the queue which contain a given string (first parameter)
#!/bin/bash if [ "$1"=="" ] then echo'Cannot delete with empty string' else grep -lir "$1" /var/spool/exim/input/ | sed -e 's/^.*\/\([a-zA-Z0-9-]*\)-[DH]$/\1/g'| xargs /usr/sbin/exim -Mrm fi
Get a count of emails in the queue per sender (sender email address is supplied by sender and can be faked)
On Android, there is a primary thread which runs all UI stuff. If a GUI operation is ran in a different thread, it just won't work, and may throw an error. If you block this thread with too much processing... well... bad things happen. Due to this design, you have to push all UI operations to this main thread using Looper.run().
Runnables pushed to this thread are always ran in FIFO execution order, which is a useful guarantee for programming.
So I decided to get smart and create the following function to add asynchronous calls that needed to be run on the primary thread. It takes a Runnable and either runs it immediately, if already on the Primary thread, or otherwise adds it to the Primary Thread’s queue.
//Run a function on primary thread publicstaticvoid RunOnPrimary(RunnableR) { LooperL=MainActivity.getMainLooper(); //Start commenting here so that items are always added to the queue, forcing in-order processesing if(Looper.myLooper()==Looper.getMainLooper()) R.run(); else //End commenting here newHandler(Looper.getMainLooper()).post(R); }
I was getting weird behaviors though in a part of the project where some actions pulled in from JavaScript were happening before subsequent actions.After the normal debugging one-by-one steps to figure it out, I realized that MAYBE some of the JavaScript calls were, for some bizarre reason, already running on the primary thread. In this case they would run immediately, before the queued items started coming in. This turned out to be the case, so I ended up having to comment out the first 3 lines after the function’s first comment (if/R.run/else), and it worked great.
I found it kind of irritating that I had to add actions to the queue when it could have been run immediately on the current thread, but oh well, I didn’t really have a choice if I want to make sure everything is always run in order across the system.
Email contact has been removed from the Contact page, in lieu of using the forum. Explanation is on the contact page.
I have removed all the projects from my projects page and site map that I will never finish and aren’t far enough along to be of any use to anyone. Most of these were written before I was 13 (in 1997). I had originally intended this site to be a collection of everything I have ever started and done, but I now see a lot of this as useless fluff. Projects removed are:
Picture Encrypt: This was just an image steganography encrypter using the picture as a one time pad. I made this as a kid before I knew image steganography and one-time-pads were things that were already known.
WebRoute: This was a Windows file system driver that allowed grabbing files from different directories in tiers. Its functionality is easily matched with symbolic links and apache redirects.
FileSync: A visually interactive rsync type program. Rsync is a better solution anyways, and I’m sure frontends for it are out there.
File Functions: Old C++ CLI libraries that are very outdated and all have better alternatives out there
Web Rich Text Editor: While I was way ahead of the curve on making this, I never polished and released it. There are so many out there nowadays anyways...
Midi: This was the very first “large” project I built as a kid (Probably ~11). Essentially, I wanted to create a competitor to what was then the only solution (I think) for creating midis/sheet-music, Cakewalk Studio.
I was recently trying to upload a TV series into Plex and was having a bit of a problem with the file naming. While I will leave the show nameless, let’s just say it has a magic dog.
Each of the files (generally) contained 2 episodes and were named S##-E##-E## (Season #, First Episode #, Second Episode #). Plex really didn’t like this, as for multi-episode files, it only supports the naming convention of first episode number THROUGH a second episode number. As an example S02-E05-E09 is considered episodes 5 through 9 of season 2. So I wrote a quick script to fix up the names of the files to consider each file only 1 episode (the first one), and then create a second symlinked file, pointing to the first episode, but named for the second episode.
So, for the above example, we would get 2 files with the exact same original filenames, except with the primary file having “S02E05,E09” in place of the episode number information, and the linked file having “S02E09-Link” in its place.
The following is the bash code for renaming/fixing a single episode file. It needs to be saved into a script file. This requires perl for regular expression renaming.
#Get the file path info and updated file name FilePath=`echo"$1"| perl -pe 's/\/[^\/]*$//g'` FileName=`echo"$1"| perl -pe 's/^.*\///g'` UpdatedFileName=`echo"$FileName"| perl -pe 's/\b(S\d\d)-(E\d\d)-(E\d\d)\b/$1$2,$3/g'`
#If the file is not in the proper format, exit prematurely if [ "$UpdatedFileName"=="$FileName" ];then echo"Proper format not found: $FilePath/$FileName" exit 1 fi
#Rename the file cd"$FilePath" mv "$FileName""$UpdatedFileName"
#Create a link to the file with the second episode name NewLinkName=`echo"$FileName"| perl -pe 's/\b(S\d\d)-(E\d\d)-(E\d\d)\b/$1$3-Link/g'` ln -s "$UpdatedFileName""$NewLinkName"
If you save that to a file named “RenameShow.sh”, you would use this like “find /PATH/ -type f -print0 | xargs -0n 1 ./RenameShowl.sh”. For windows, make sure you use windows symlinks with /H (to make them hard file links, as soft/symbolic link files really just don’t work in Windows).
Continuing my attempt to get stuff moved over to GitHub, I just added a GitHub Repo for HackPics.
This is a rather old project that I originally created in late 2004. It had some minor modifications done to it in mid 2008 to help with the code clarity so others could work with it easier. I was requested to add it to GitHub a few weeks back, so here it is. Strangely enough, this has always been one of my most popular Projects. Always seems to happen with video game utilities/projects.
The rather long explanation of the project history and my findings regarding this project can be found in the request thread on my forum.
Just uploaded a new project (that I actually got up on github a bit over a week ago), HTTP-Forwardersv1.0.0. It completely mirrors a website on a different domain by forwarding through a server. It has 2 HTTP forwarder/proxies. One in Go, and the other in PHP. Note: The Go version is much more advanced, reliable, and fast.
