One of the companies I work for recently moved one of our cPanel servers to a new collocation, still running cPanel. We decided to use a new backup solution called r1soft, which so far has been working spectacularly. I’d love to use it for my personal computers, except the licenses, which are geared towards enterprise business, are way too costly.
However, since r1soft only backs up files (on the incrementally block level, yay) you can’t use it to restore a cPanel account. It can only restore things like the user’s home directory and SQL databases. Because of this, when we had need to restore an entire account today, and found out there is no easy/quick way to do it, we were up a creek. The obvious future solution for this would be to use cPanel’s backup (or legacy backup) systems, but unfortunately, you can’t easily set them to not backup the user’s databases and home directory, which can be very large, and are already taken care of by r1soft. I ended up adding the following script, ran nightly via cron, to back up user account settings.
It saves all the user settings under the backup path in their own directory, uncompressed, and named cpmove-USERNAME. It is best to do it this way so r1soft’s incremental backups don’t have much extra work if anything changes. Make sure to change line 3 in the following script to the path where you want backups to occur.
#!/bin/bash #Create and move to backup directory BACKUPDIR=/backup/userbackup mkdir -p $BACKUPDIR#Make sure the directory exists cd$BACKUPDIR
#Extract from and remove the tar container file tar -xvf cpmove-$USER.tar rm -f cpmove-$USER.tar
#Save MySQL user settings mysqldump --compact -fnt -w "User LIKE '$USER""_%'" mysql user db tables_priv columns_priv procs_priv proxies_priv \ | perl -pe "s~('|NULL)\),\('~\1),\n('~ig" \ > cpmove-$USER/mysql-users.sql done;
This script skips a few backup items that need to be noted. Mailman, logs, homedir, and bandwidth data should all be easy 1:1 copy over restores from r1soft. I excluded them because those can take up a lot of room, which we want r1soft to handle. The same goes for MySQL, except that your MySQL users are not backed up to your account, which is why I added the final section.
Do note, for the final section, the line starting with “| perl” is optional. It is there to separate the insert rows into their own lines. A very minor warning though; it would also pick up cases where the last field in MySQL’s user table ends in “NULL),(”. This would only happen if someone is trying to be malicious and knew about this script, and even then, it couldn’t harm anything.
Bonus note: To restore a MySQL database which does not use a shared-file (like InnoDB does by default), you could actually stop the MySQL server, copy over the binary database files, and start the server back up.
AppDirectory='/cygdrive/c/Users/Administrator/AppData/Local/Plex Media Server/'; sqlite3 "$AppDirectory/Plug-in Support/Databases/com.plexapp.plugins.library.db" 'SELECT file, title, MDI."index", hints FROM media_parts AS MP INNER JOIN media_items AS MI ON MI.id=MP.media_item_id INNER JOIN metadata_items AS MDI ON MDI.id=MI.metadata_item_id;'[/code
When I first created my website 10 years ago, from scratch, I did not want to deal with writing a comment system with HTML markups. And in those days, there weren’t plugins for everything like there is today. My solution was setting up a forum which would contain a topic for every Project, Update, and Post, and have my pages mirror the linked topic’s posts.
I had just put in a quick hack at the time in which the pulled SMF message’s body had links converted from bbcode (there might have been 1 other bbcode I also hooked). I had done this with regular expressions, which was a nasty hack.
So anywho, I finally got around to writing a script that converts SMF messages’ bbcode to HTML and caches it. You can download it here, or see the code below. The script is optimized so that it only ever needs to load SMF code when a post has not yet been cached. Caching happens during the initial loading of an SMF post within the script’s main function, and is discarded if the post is changed.
The script requires that you run the query on line #3 of itself in your SMF database. Directly after that are 3 variables you need to set. The script assumes you are already logged in to the appropriate user. To use it, call “GFTP\GetForumTopicPosts($ForumTopicID)”. I have the functions split up so you can do individual posts too if needed (requires a little extra code).
<? //This SQL command must be ran before using the script //ALTER TABLE smf_messages ADD body_html text, ADD body_md5 char(32) DEFAULT NULL;
functionGetForumTopicPosts($ForumTopicID) { //Change to the forum database global$ForumInfo; $CurDB=mysql_fetch_row(mysql_query('SELECT database()'))[0]; if($CurDB!=$ForumInfo['DBName']) mysql_select_db($ForumInfo['DBName']); $OldEncoding=SetEncoding(true);
//Get the posts $PostsInfos=Array(); $PostsQuery=mysql_query('SELECT'.implode(', ', PostFields())." FROM $ForumInfo[MessageTableName] WHERE id_topic='".intval($ForumTopicID). "' AND approved=1 ORDER BY id_msg ASC LIMIT 1, 9999999"); if($PostsQuery) //If query failed, do not process while(($PostInfo=mysql_fetch_assoc($PostsQuery)) && ($PostsInfos[]=$PostInfo)) if(md5($PostInfo['body'])!=$PostInfo['body_md5']) //If the body md5s do not match, get new value, otherwise, use cached value ProcessPost($PostsInfos[count($PostsInfos)-1]); //Process the lastest post as a reference
//Restore from the forum database if($CurDB!=$ForumInfo['DBName']) mysql_select_db($CurDB); SetEncoding(false, $OldEncoding);
//Return the posts return$PostsInfos; }
functionProcessPost(&$PostInfo) //PostInfo must have fields id_msg, body, body_md5, and body_html { //Load SMF global$ForumInfo; if(!defined('SMF')) { global$context; require_once(rtrim($ForumInfo['Location'], DIRECTORY_SEPARATOR).DIRECTORY_SEPARATOR.'SSI.php'); mysql_select_db($ForumInfo['DBName']); SetEncoding(); }
//Update the cached body_html field $ParsedCode=$PostInfo['body_html']=parse_bbc($PostInfo['body']); $EscapedHTMLBody=mysql_escape_string($ParsedCode); $BodyMD5=md5($PostInfo['body']); mysql_query("UPDATE$ForumInfo[MessageTableName] SET body_html='$EscapedHTMLBody', body_md5='$BodyMD5'WHERE id_msg=$PostInfo[id_msg]"); }
//The fields to select in the Post query functionPostFields() { returnArray('id_msg', 'poster_time', 'id_member', 'subject', 'poster_name', 'body', 'body_md5', 'body_html'); }
//Swap character encodings. Needs to be set to utf8 functionSetEncoding($GetOld=false, $NewSet=Array('utf8', 'utf8', 'utf8')) { //Get the old charset if required $CharacterVariables=Array('character_set_client', 'character_set_results', 'character_set_connection'); $OldSet=Array(); if($GetOld) { //Fill in variables with default in case they are not found foreach($CharacterVariablesas$Index=>$Variable) $OldSet[$Variable]='utf8';
//Query for the character sets and update the OldSet array $Query=mysql_query('SHOW VARIABLES LIKE "character_%"'); while($VariableInfo=mysql_fetch_assoc($Query)) if(isset($OldSet[$VariableInfo['Variable_name']])) $OldSet[$VariableInfo['Variable_name']]=$VariableInfo['Value'];
$OldSet=array_values($OldSet); //Turn back into numerical array }
//Change to the new database encoding $CompiledSets=Array(); foreach($CharacterVariablesas$Index=>$Variable) $CompiledSets[$Index]=$CharacterVariables[$Index].'="'.mysql_escape_string($NewSet[$Index]).'"'; mysql_query('SET '.implode(', ', $CompiledSets));
//If requested, return the previous values return$OldSet; } ?>
Amazon EC2 is a great resource for cheap virtual servers to do simple things, like DNS or (low bandwidth) VPNs. I had the need this morning to set up a DNS server for a company which needed to blacklist a list of domains. The simplest way to do this is by editing all the computers’ hostfiles, but that method leaves a lot to be desired. Namely, blocking entire domains (as opposed to single subdomains), and deploying changes. Centralizing in a single place makes the job instant, immediate, and in the end, faster.
The following are the steps I used to set this up on an EC2 server. All command line instructions are followed by a single command you can run to execute the step. There is a full script below, at the end of the post, containing all steps from when you first login to SSH ("Login to root") to the end.
I am not going to go into the details of setting up an EC2 instance, as that information can be found elsewhere. I will also be skipping over some of the more obvious steps. Just create a default EC2 instance with the “Amazon Linux AMI”, and I will list all the changes that need to be made beyond that.
Creating the instance
For the first year, for the instance type, you might as well use a t2.micro, as it is free. After that, a t2.nano (which is a new lower level) currently at $56.94/year ($0.0065/Hour), should be fine.
After you select your instance type, click “Review and Launch” to launch the instance with all of the defaults.
After the confirmation screen, it will ask you to create a key pair. You can see other tutorials about this and how it enables you to log into your instance.
Edit the security group
Next, you need to edit the security group for your instance to allow incoming connections.
Go to “Instances” under the “Instances” group on the left menu, and click your instance.
In the bottom of the window, in the “Descriptions” tab, click the link next to “Security Groups”, which will bring you to the proper group in the security groups tab.
Right click it and “Edit inbound Rules”.
Make sure it has the following rules with Source=Anywhere: ALL ICMP [For pinging], SSH, HTTP, DNS (UDP), DNS (TCP)
Assign a permanent IP to your instance
To do this, click the “Elastic IPs” under “Network & Security” in the left menu.
Click “Allocate New Address”.
After creating it, right click the new address, then “Associate Address”, and assign it to your new instance.
You should probably set this IP up as an A record somewhere. I will refer to this IP as dns.yourdomain.com from now on.
Login to root
SSH into your instance as the ec2-user via “ssh ec2-user@dns.yourdomain.com”. If in windows, you could also use putty.
Sudo into root via “sudo su”.
Allow root login
At this point, I recommend setting it up so you can directly root into the server. Warning: some people consider this a security risk.
Copy your key pair(s) to the root user via “cat /home/ec2-user/.ssh/authorized_keys > /root/.ssh/authorized_keys”
Set SSHD to permit root logins by changing the PermitRootLogin variable to “yes” in /etc/ssh/sshd_config. A quick command to do this is “perl -pi -e 's/^\s*#?\s*PermitRootLogin.*$/PermitRootLogin yes/igm' /etc/ssh/sshd_config”, and then reload the SSHD config with “service sshd reload”. Make sure to attempt to directly log into SSH as root before exiting your current session to make sure you haven’t locked yourself out.
Install apache (the web server), bind/named (the DNS server), and PHP (a scripting language)
yum -y install bind httpd php
Start and set services to run at boot
service httpd start; service named start; chkconfig httpd on; chkconfig named on;
Set the DNS server to be usable by other computers
Edit /etc/named.conf and change the 2 following lines to have the value “any”: “listen-on port 53” and “allow-query”
perl -pi -e 's/^(\s*(?:listen-on port 53|allow-query)\s*{).*$/$1 any; };/igm' /etc/named.conf; service named reload;
Point the DNS server to the blacklist files
This is done by adding “include "/var/named/blacklisted.conf";” to /etc/named.conf
Put the following into /var/named/blacklisted.db . Make sure to change dns.yourdomain.com to your domain (or otherwise, “localhost”), and 1.1.1.1 to dns.yourdomain.com’s (your server’s) IP address. Make sure to keep all periods intact.
$TTL 14400 @ IN SOA dns.yourdomain.com. dns.yourdomain.com ( 2003052800 86400 300 604800 3600 ) @ IN NS dns.yourdomain.com. @ IN A 1.1.1.1 * IN A 1.1.1.1
The first 2 lines tell the server the domains belong to it. The 3rd line sets the base blacklisted domain to your server’s IP. The 4th line sets all subdomains of the blacklisted domain to your server’s IP.
This can be done via (Update the first line with your values)
YOURDOMAIN="dns.yourdomain.com"; YOURIP="1.1.1.1"; echo -ne "\$TTL 14400\n@ IN SOA $YOURDOMAIN. $YOURDOMAIN ( 2003052800 86400 300 604800 3600 )\n@ IN NS $YOURDOMAIN.\n@ IN A $YOURIP\n* IN A $YOURIP" > /var/named/blacklisted.db;
Fix the permissions on the blacklist files
chgrp named /var/named/blacklisted.*; chmod 660 /var/named/blacklisted.*;
Set the server’s domain resolution name servers
The server always needs to look at itself before other DNS servers. To do this, comment out everything in /etc/resolv.conf and add to it “nameserver localhost”. This is not the best solution. I’ll find something better later.
At this point, it’s a good idea to make sure the DNS server is working as intended. So first, we’ll add an example domain to the DNS server.
Add the following to /var/named/blacklisted.conf and restart named to get the server going with example.com: “zone "example.com" { type master; file "blacklisted.db"; };”
echo 'zone "example.com" { type master; file "blacklisted.db"; };' >> /var/named/blacklisted.conf; service named reload;
Ping “test.example.com” and make sure it’s IP is your server’s IP
Set your computer’s DNS to your server’s IP in your computer’s network settings, ping “test.example.com” from your computer, and make sure the returned IP is your server’s IP. If it works, you can restore your computer’s DNS settings.
Have the server return a message when a blacklisted domain is accessed
Add your message to /var/www/html
echo 'Domain is blocked' > /var/www/html/index.html
Set all URL paths to show the message by adding the following to the /var/www/html/.htaccess file
Turn on AllowOverride in the /etc/httpd/conf/httpd.conf for the document directory (/var/www/html/) via “ perl -0777 -pi -e 's~(<Directory "/var/www/html">.*?\n\s*AllowOverride).*?\n~$1 All~s' /etc/httpd/conf/httpd.conf”
Start the server via “service httpd graceful”
Create a script that allows apache to refresh the name server’s settings
Create a script at /var/www/html/AddRules/restart_named with “/sbin/service named reload” and set it to executable
Allow the user to run the script as root by adding to /etc/sudoers “apache ALL=(root) NOPASSWD: /var/www/html/AddRules/restart_named” and “Defaults!/var/www/html/AddRules/restart_named !requiretty”
//Output to the file $FinalDomainData=Array(); foreach($FinalDomainListas$Domain) $FinalDomainData[]= "zone \"$Domain\" { type master; file \"blacklisted.db\"; };"; file_put_contents($BlockedFile, implode("\n", $FinalDomainData));
//Reload named print`sudo /var/www/html/AddRules/restart_named`; ?>
Add the “apache” user to the “named” group so the script can update the list of domains in /var/named/blacklisted.conf via “usermod -a -G named apache; service httpd graceful;”
Run the domain update script
To add a domain (separate by commas): http://dns.yourdomain.com/AddRules/?Domains=domain1.com,domain2.com
To remove a domain (add “Remove&” after the “?”): http://dns.yourdomain.com/AddRules/?Remove&Domains=domain1.com,domain2.com
To list the domains: http://dns.yourdomain.com/AddRules/?List
Warning: Putting the password file in an http accessible directory is a security risk. I just did this for sake of organization.
Create the user+password via “htpasswd -bc /var/www/html/AddRules/.htpasswd USERNAME” and then entering the password
[Edit on 2016-01-30 @ noon]
To permanently set “localhost” as the resolver DNS, add “DNS1=localhost” to “/etc/sysconfig/network-scripts/ifcfg-eth0”. I have not yet confirmed this edit.
Security Issue
Soon after setting up this DNS server, it started getting hit by a DNS amplification attack. As the server is being used as a client’s DNS server, turning off recursion is not available. The best solution is to limit the people who can query the name server via an access list (usually a specific subnet), but that would very often not be an option either. The solution I currently have in place, which I have not actually verified if it works, is to add a forced-forward rule which only makes external requests to the name server provided by Amazon. To do this, get the name server’s IP from /etc/resolv.conf (it should be commented from an earlier step). Then add the following to your named.conf in the “options” section.
forwarders { DNS_SERVER_IP; }; forward only;
After I added this rule, external DNS requests stopped going through completely. To fix this, I turned “dnssec-validation” to “no” in the named.conf. Don’t forget to restart the service once you have made your changes.
Make sure to run this as root (login as root or sudo it)
Download the script here. Make sure to chmod and sudo it when running. “chmod +x dnsblacklist_install.sh; sudo ./dnsblacklist_install.sh;”
#User defined variables VARIABLES_SET=0; #Set this to 1 to allow the script to run YOUR_DOMAIN="localhost"; YOUR_IP="1.1.1.1"; BLOCKED_ERROR_MESSAGE="Domain is blocked"; ADDRULES_USERNAME="YourUserName"; ADDRULES_PASSWORD="YourPassword";
#Confirm script is ready to run if [ $VARIABLES_SET != 1 ];then echo'Variables need to be set in the script'; exit 1; fi if [ `whoami`!='root' ];then echo'Must be root to run script. When running the script, add "sudo" before it to' \ 'run as root'; exit 1; fi
#Install services yum -y install bind httpd php; chkconfig httpd on; chkconfig named on; service httpd start; service named start;
#Set the DNS server to be usable by other computers perl -pi -e 's/^(\s*(?:listen-on port 53|allow-query)\s*{).*$/$1 any; };/igm' \ /etc/named.conf; service named reload;
#Create the blacklist zone file echo -ne "\$TTL 14400 @ IN SOA $YOUR_DOMAIN. $YOUR_DOMAIN ( 2003052800 86400 300 604800 3600 ) @ IN NS $YOUR_DOMAIN. @ IN A $YOUR_IP * IN A $YOUR_IP"> /var/named/blacklisted.db;
#Fix the permissions on the blacklist files chgrp named /var/named/blacklisted.*; chmod 660 /var/named/blacklisted.*;
#Set the server’s domain resolution name servers perl -pi -e 's/^(?!;)/;/gm' /etc/resolv.conf; echo -ne '\nnameserver localhost'>> /etc/resolv.conf;
#Run a test echo'zone "example.com" { type master; file "blacklisted.db"; };'>> \ /var/named/blacklisted.conf; service named reload; FOUND_IP=`dig -t A example.com | grep -ioP "^example\.com\..*?"'in\s+a\s+[\d\.:]+'| \ grep -oP '[\d\.:]+$'`; if [ "$YOUR_IP"=="$FOUND_IP" ] then echo'Success: Example domain matches your given IP'> /dev/stderr; else echo'Warning: Example domain does not match your given IP'> /dev/stderr; fi
#Have the server return a message when a blacklisted domain is accessed echo"$BLOCKED_ERROR_MESSAGE"> /var/www/html/index.html; perl -0777 -pi -e 's~(<Directory "/var/www/html">.*?\n\s*AllowOverride).*?\n~$1 All~s' \ /etc/httpd/conf/httpd.conf; echo -n 'RewriteEngine on RewriteCond %{REQUEST_URI} !index.html RewriteCond %{REQUEST_URI} !AddRules/ RewriteRule ^(.*)$ /index.html [L]'> /var/www/html/.htaccess; service httpd graceful;
#Create a script that allows apache to refresh the name server’s settings mkdir /var/www/html/AddRules; echo'/sbin/service named reload'> /var/www/html/AddRules/restart_named; chmod 755 /var/www/html/AddRules/restart_named;
#Create a script that allows the user to add, remove, and list the blacklisted domains echo -n $'<?php //Get old domains $BlockedFile=\'/var/named/blacklisted.conf\'; $CurrentZones=Array(); foreach(explode("\\n", file_get_contents($BlockedFile)) as $Line) if(preg_match(\'/^zone "([\\w\\._-]+)"/\', $Line, $Results)) $CurrentZones[]=$Results[1];
//Output to the file $FinalDomainData=Array(); foreach($FinalDomainList as $Domain) $FinalDomainData[]="zone \\"$Domain\\" { type master; file \\"blacklisted.db\\"; };"; file_put_contents($BlockedFile, implode("\\n", $FinalDomainData));
//Reload named print `sudo /var/www/html/AddRules/restart_named`; ?>'> /var/www/html/AddRules/index.php;
usermod -a -G named apache; service httpd graceful;
objdump -p $1 | grep -Pzo '(?is)^\[Ordinal/Name Pointer\] Table.*?\n\n' | grep -oP '(?<=\d\] ).*$'If you saved the script as "get_dll_exports", to run it against multiple DLLs at a time, create another script as follows
for i in $@; do echo -e "--------\n$i\n--------"; get_dll_exports $i; doneOr to process multiple dlls, but output all of a single file's results on one line with the filename preceding
Here is a simple bash script (hereby referred to as “Propagate.sh”) which syncs /var/www/html/ to all of your slave instances. It uses the “aws” command line interface provided by Amazon, which comes default with the Amazon Linux starter AMI.
#The first command line of the script contains the master’s IP, so it does not sync with itself. export LocalIP=Your_Master_IP_Here;
#Get the IPs of all slave instances export NewIPs=`aws ec2 describe-instances | grep '"PrivateIpAddress"' | perl -i -pe 's/(^.*?: "|",?\s*?$)//gm' | sort -u | grep -v $LocalIP`
#Loop over all slave instances for i in $NewIPs; do echo "Syncing to: $i"; #Run an rsync from the master to the slave rsync -aP -e 'ssh -o StrictHostKeyChecking=no' /var/www/html/ root@$i:/var/www/html/; done
You may also want to add “-o UserKnownHostsFile=/dev/null” to the SSH command (directly after “-o StrictHostKeyChecking=no”), as a second EC2 instance may end up having the same IP as a previously terminated instance. Another solution to that problem is syncing the “/etc/ssh/ssh_host_rsa_key*” from the master when an instance initializes, so all instances keep the same SSH fingerprint.
To let other people manually execute this script, you can create a PHP file with the following in it. (Change /var/www/ in all below examples to where you place your Propagate.sh)
If your Propagate.sh needs to be ran as root, which it may if your PHP environment is not run as the user root (usually “apache”), then you need to make sure it CAN run as root without intervention. To do this, add the following to the /etc/sudoers file
Change the user from “apache” to the user which PHP runs as (when running through apache). I included “whoami” as a valid sudoer application for testing purposes. Also, in the sudoers file, if “Defaults requiretty” is turned on, you will need to comment it/turn it off.
While I did not mention it in yesterday's post, I thought I should at least mention it here. There are other ways to keep file systems in sync with each other. This is just a good use case for when you want to keep all instances as separate independent entities. Another solution to many of the previously mentioned problems is using Amazon's new EFS, which is currently still in preview mode.
