The Great Zero Challenge: A challenge to confirm whether or not a professional data recovery firm or any individual(s) or organization(s) can recover data from a hard drive that has been overwritten with zeros once. All they used is the Unix dd command using /dev/zero as input to overwrite the drive.

They are doing this because many people believe that in order to permanently delete data from a modern hard drive that multiple overwrites with random data, mechanical grinding, degaussing and incinerating must be used and even physically destroy them just to be extra safe.

As far as challenges go, this is one that many will pass, because no data recovery firm claims to be able to do zero recovery other than those with access to electron microscopy tools.

Many professional recovery firms for most part use tools that scan through the partition and file table area and perhaps even the entire disk to locate data that has either been marked erased or had references removed (for a full disk scan) and then restoring it. Perhaps they’ll also move the spindle from a dead drive into a new case to complete the operation, but I doubt there are many companies that will actually do electron force microscopy for you and even fewer that will do it at anything other than an astronomical fee.

My opinion is that the recovery business is a focus around confidence that a professional will be doing the recovery and that you or your employees won’t worsen the situation. In the event that a drive with critical data fails and you don’t have a backup, who wants to be the person responsible for damaging the disk during recovery?

Anyway, this whole debate should be moot by now. If you want to secure your drive use full disk encryption (now freely available in TrueCrypt) and when it comes to destroying the data just overwrite the header area a thousand times with random garbage. It will take only a second or two, and the whole drive will be useless to anyone. Oh, while you are at it, why not use the dd command these guys used.

Of course it would also be nice if more manufacturers were producing encrypted disks as standard with verified schemes so that we wouldn’t have to do encryption in software.

The Great Zero Challenge: http://16systems.com/zero/index.html

Besides manual security hardening of a Linux OS, let’s check out a free open-source tool to automate and simplify the process. Bastille will disable unnecessary services and install operating system updates as well as configure a firewall, enforce password policies, create a second root-level account and more. What’s nice is that Bastille leads the user through a simple series of yes/no questions, giving a detailed explanation of why each question is asked and what will happen if ‘yes’ is chosen. It doesn’t merely expect guesswork, nor does it blindly alter your system – instead, it genuinely hardens your computer and educates on security in the process.

The good thing is that you’re also not locked in to Bastille’s changes should you decide some of the setting changes weren’t for you. Running RevertBastille automatically restores the state of all config files and settings to just how they were before Bastille made any changes. Obviously, if you make changes to your system manually after running Bastille, you will lose these too so it is best to test changes as soon as possible after applying to ensure you won’t harm anything else if you need to revert.

Unfortunately, Bastille is not for everyone: versions exist for Red Hat, SUSE, Debian, Gentoo and Mandrake (as well as non-Linux UNIX variants HP-UX and MacOS X). If you do run one of those systems, you really are well-advised to run Bastille. You can download the latest version from SourceForge.

Let’s give Bastille a run-through.
Launch Bastille by calling up a terminal prompt as root and executing ./InteractiveBastille. You are lead through a series of security steps, as follows.

1.  Apply a firewall to prevent access to potentially vulnerable services, using iptables. This is a big topic which could not be adequately covered here. Fortunately, Bastille’s explanations do an admirable job. In one sense, this is redundant; if the service has been disabled as we discussed above, there won’t be anything listening on the port which can be exploited. However, you might later restore a service for testing or for internal use. Or it may be restored inadvertently. Whatever the reason, Bastille errs on the side of tougher security by protecting your system from the same exploits via more than one method.

Retrieve and apply available operating system patches, as discussed above.

2. Audit the system tools which have the SUID flag set and which run as the superuser, even for ordinary users. The danger of SUID apps is they perform actions with full superuser powers no matter who executes them. This is essential in some cases: for instance, if the passwd command couldn’t write back to the shadowed password file then nobody could actually change their password. However, you may not want ordinary users running the dump and restore commands, both of which come with SUID status out-of-the-box.

3. Tighten up account security. Here, Bastille first asks to create a second account with root-level access. This means you can disable root if desired, or at the very least if you exclusively use the second account, you can tell if someone else is trying to log in as root because you know it won’t be you. This section of Bastille also prompts to enforce password aging and some other items like assigning a restricted or useless shell to non-user accounts. There’s wisdom in this last point.

4. Enhance boot security. This helps restrict the computer even if someone can get physical access to it and try starting it up in single-user mode.

5. Deactivate or restrict unnecessary services, as discussed above.
From this point, the remaining modules are less significant (though still beneficial) and include disabling program compilation, limiting system usage, increasing logging, installing SSH, tightening up DNS and Apache, disabling printing and a couple of other things.

Bastille now exits, but has not yet made any changes. All your choices have been saved to a configuration file. Run ./BackEnd.pl to actually enforce them. Reboot and test out your hardened server. Any malicious attackers will find far less vulnerabilities and options against your computer.

Security is something we all need to take seriously. Many people may not even be aware that they have possible insecurities. Fortunately, the above steps are easy to understand and simple to implement.

[source: http://www.itwire.com/index2.php?option=com_content&do_pdf=1&id=13976]

Sometimes when running through a CSV or any kind of a log file, you may encounter lists with a lot of duplicates. I will show an example of the simplest order here.

Say, you have a duplicates.txt that goes

one
two
three
one
four
two
four

Now, how to remove duplicates from a list such as the one shown above. If you use a command such as

sort – u < duplicates.txt or cat duplicates.txt | sort | uniq

you may end up with a list that while stripping out the duplicates, does not keep the original order

four
two
three
one

Now, there is a way where you can remove duplicates from a list… but still keep the original order.

nl duplicates.txt | sort -k2 -u | sort -n | cut -f2-

Step 1. First number the entries in the duplicates.txt using nl

nl duplicates.txt

This will give you the list:

1 one
2 two
3 three
4 one
5 four
6 two
7 four

Step 2. We now need to sort the list

nl duplicates.txt | sort -k2

7 four
5 four
1 one
4 one
3 three
2 two
6 two

Step 3. Now, we need to remove the lines with duplicate fields:

nl duplicates.txt | sort -k2 -u

7 four
1 one
3 three
2 two

Step 4. Restore the original order:

nl duplicates.txt | sort -k2 -u | sort -n

1 one
2 two
3 three
7 four

Step 5. Remove the numbering we inserted in Step 1

nl duplicates.txt | sort -k2 -u | sort -n | cut -f2-

one
two
three
four

A lot of us who use Linux at work/school or have always grown up using Unix commands and using the Unix shell for years and more often than not, there are instances where a ls command comes more naturally than the dir command at the command prompt when using Windows.

For the most part, a lot of us work around this drawback using the excellent tool: Cygwin. Cygwin is available for windows users here.The Cygwin tools are ports of the popular GNU development tools for Microsoft Windows. They run thanks to the Cygwin library which provides the UNIX system calls and environment these programs expect.

With these tools installed, it is possible to write Win32 console or GUI applications that make use of the standard Microsoft Win32 API and/or the Cygwin API. As a result, it is possible to easily port many significant Unix programs without the need for extensive changes to the source code. This includes configuring and building most of the available GNU software . Even if the development tools are of little to no use to you, you may have interest in the many standard Unix utilities provided with the package. They can be used both from the bash shell (provided) or from the standard Windows command shell.

While Cygwin would be an obvious choice for many Unix/Linux power users, there is an excellent and a much simpler alternative to using Cygwin. In this article, I will show you how to run your Unix commands right in the windows command prompt.

For this, we will be using CoreUtils. CoreUtils is a collection of basic file, shell and text manipulation utilities of the GNU operating system. These are the core utilities which are expected to exist on every OS. And when I talk about File utilities, they include chgrp, chmod, cp, dd, du, ln, ls, mkdir, mv, rm, touch, vdir among others. A sample of the text utilities include cat, cksum, cut, join, md5sum, shasum, sort, split etc. The shell root commands include echo, chroot, hostname, nice, pathchk, tty, who, whoami and yes su. So it is pretty much the whole nine yards here… The direct link for download of the CoreUtils package available through SourceForge is available here.

Once installed, you will need to add the path to the utilities to your PATH environment variable. Follow the steps below to achieve your Unix Shell for Windows

1. Click on Start –> Run and enter sysdm.cpl to bring up the system properties Dialog

2. Click on the Advanced tab –> Environment variables button

3. In the System Variables pane, scroll down to Path and then click on edit.

4. Under Edit System Variable, in the variable value, at the end of the line , type the following including the semicolon which separates the individual elements in the path variable. ;C:\Program Files\GnuWin32\bin

Congratulations !! You have now added the GNUWin directory to your path and Unix commands can now be executed directly from the command line and run natively on the Win32 command prompt without the need for any emulation layer as shown below using the example of dir vs ls

Downloads and Sources

1. Download CYGWIN

2. Download CoreUtils