pages tagged filesystemBitPipelinehttp://www.bitpipeline.eu/tags/filesystem/BitPipelineikiwiki2013-09-23T09:24:57ZAutomaticaly publishing a websitehttp://www.bitpipeline.eu/Blog/20120601-Automagicaly_publishing_a_website/2013-09-23T09:24:57Z2012-06-01T06:24:00Z
<p>Most of the Web hosting companies provide a FTP to publish the set of files that form the website. FTP is a really good way of transferring files between two hosts: it's simple and fast. Unfortunately that simplicity is also it Achilles heel when it comes to publishing a web site you're creating off line. The problem is that you would have to keep track of the file changes, transfer the changed files and remove any file that was deleted locally. Too much trouble for me.</p>
<p>The solution, or at least a solution, is a based on two concepts:</p>
<ul>
<li>mounting the remote ftp site in the local host;</li>
<li>syncing the locally created Web site to the remote mounted file system.</li>
</ul>
<p>To mount the remote file system I use <a href="http://curlftpfs.sourceforge.net/">curlftpfs</a>, a filesystem based on <a href="http://fuse.sourceforge.net">FUSE</a> which implements a user level file system over FTP.</p>
<p>After having the remote filesystem mouted in the local host it is easy to synchronise it using the good old <a href="http://rsync.samba.org/">rsync</a> although I had to tell <code>rsync</code> to ignore errors that show up when it tries to change the time stamp of the remote files.</p>
<p>As I like to automate everything I created a Makefile with the following targets:</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><font color="#000000"> 1:</font> <font color="#990000">mount:</font>
<font color="#000000"> 2:</font> curlftpfs -o tlsv1 -o <font color="#009900">user</font><font color="#990000">=</font>USER FTP_HOST LOCAL_MOUNT_DIR
<font color="#000000"> 3:</font>
<font color="#000000"> 4:</font> <font color="#990000">umount:</font>
<font color="#000000"> 5:</font> fusermount -u LOCAL_MOUNT_DIR
<font color="#000000"> 6:</font>
<font color="#000000"> 7:</font> <font color="#990000">publish:</font> <font color="#990000">~</font>/mnt/bitpipeline/web
<font color="#000000"> 8:</font> rsync -recursive -verbose --progress --delete-during --ignore-errors LOCAL_WEB_DIR LOCAL_MOUNT_DIR/REMOTE_WEB_DIR</tt></pre>
<p>Where:
<code>USER</code>
: is the ftp user name</p>
<p><code>FTP_HOST</code>
: is the remote ftp address to use</p>
<p><code>LOCAL_MOUNT_DIR</code>
: is the directory where the remote ftp site will be mounted</p>
<p><code>LOCAL_WEB_DIR</code>
: is the directory were the Web site is built (in the local host)</p>
<p><code>REMOTE_WEB_DIR</code>
: is the remote directory that is root of the Web site file system</p>
<p>It might happen that your host does not support TLS, which is bad: your password will be travelling in the Net without any protection from preying eyes. If you still want to use this scheme than remove the option <code>-o tlsv1</code> from the <code>curlftpfs</code> command.</p>
<p>Then the workflow would be:</p>
<ol>
<li>run "<code>make mount</code>". You'll be asked the password for the user.</li>
<li>develop the web site (I use <a href="http://ikiwiki.info/">ikiwiki</a> to generate the site).</li>
<li>run "<code>make publish</code>". This might take some time…</li>
<li>if you have more work to do go to step 2.</li>
<li>run "<code>make umount</code>".</li>
</ol>
<p>The last step might fail if any process has any file descriptor open to the remote file system. Find which process (using <a href="ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/">lsof</a> for example) and have it close that file descriptor (this might mean terminating the process). Or you can add the "<code>-z</code>" to the <code>fusermount</code> command. That will make it do a "lazy umount", meaning that the file system will be invisible to all processes except those that have a file descriptor open to it and as soon as all file descriptors are closed the file system will be properly unmounted.</p>
<p>To have the needed tools in my development machine (using gentoo) I had to unmask <code>net-fs/curlftpfs</code>.</p>
Setting up a new gentoo linux machine: partitioninghttp://www.bitpipeline.eu/Blog/20120130-Setting_up_a_new_gentoo_linux_machine-partitioning/2013-09-23T09:24:57Z2012-01-30T01:27:00Z
<p>After eight years of service my poor old companion notebook, a Sony Vaio VGN-S2XP, was in need of retirement and a new laptop had to be bought. After searching around a bit my choice fell on a Toshiba r840-125. Good price, good video card, good feeling to the hardware and a trusty brand (you would not find me getting a HP or Dell for sure).</p>
<p>Unfortunately it's still almost impossible to buy a computer without paying homage to Microsoft Coorporation and to deliver money to their doorstep even when we have no intention of using it's faulty OS. Ok, maybe not faulty, maybe just slow, laggy, buggy, annoying, incomplete, ugly, etc. You know what I mean and if you don't then you should try to get out some more... Incredible how governments allow this small sucking of their citizens to go on. If people don't give money to M$ then the gov guys could come and get some more, no? (this is irony)</p>
<p>Well.. so setting up a new machine... I decided to leave the legacy OS installed (that windows stuff) and the rescue partition too. This way if something goes wrong with the hardware I can try it with the software that was chosen by Toshiba.</p>
<p>That lead me to the first step of setting up the computer: partitioning the hard disk.</p>
<h1 id="Partitions">Partitions</h1>
<h2 id="Keeping_the_legacy_OS_and__the_existing_partitions">Keeping the legacy OS and the existing partitions</h2>
<p>This was the very easy part. Just record a CD with the wonderful <a href="http://www.sysresccd.org/Main_Page">System Rescue CD</a>, boot it to a graphical interface and use <a href="http://gparted.sourceforge.net/">GParted</a> to do the hard work. </p>
<p>Toshiba ships this hardware with three primary partitions:</p>
<ol>
<li>Sytem, a hidden NTFS partition, a bootable partition where I guess the boot of the installed OS is</li>
<li>disk, the normal disk for the owner to use (ahhh.. the assumptions the hardware manufacturers do...)</li>
<li>HDRecovery, another hidden NTFS partition, where the data for reconstructing the original OS is</li>
</ol>
<p>The first and third partitions I left them as they were (no resizing, no moving). The second I did resize to about 10% of its original size (little more than what was already being used.</p>
<h2 id="Using_one_single_extended_partition_for_linux">Using one single extended partition for linux</h2>
<p>In my experience with gentoo I became used to have the following partition layout: </p>
<p><code>/</code>
: the root of the system, where the main OS is installed (binaries, libraries, resources, temporary stuff, etc)</p>
<p><code>/boot</code>
: the place where the binaries and disk images needed to boot linux are placed</p>
<p><code>/home</code>
: where the user home directories are located</p>
<p><code>/usr/portage</code>
: where the portage tree (the files that describe each of the available software used by the system) is placed</p>
<p>It might not be the best possible layout but it works for me. Some people like to have a specific partition for <code>/var</code> but I can't really see the point and my machines have been running happily since '94. </p>
<p>I couldn't use primary partitions for my Linux gentoo system (only one of a limit of four was available) but I could use extended partitions (think of it as partitions inside a partition). So a extended partition was created in the freed disk space (from the resizing of partition two) and then three partitions were created, one for the <code>/boot</code>, one for SWAP and one for the entire linux system... but there's a catch. The last partition created is a <a href="http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">LVM</a> partition. With a bit more effort to do the initial setup of the system I will end up having a abstraction on the storage space - if I ever feel short of space I can throw away the space reserved by Toshiba for a legacy OS and simple expand (move and resize probably) the LVM partition to use more space).</p>
<h2 id="Some_tweaks_for_the_partitioning">Some tweaks for the partitioning</h2>
<p>Most of the previous partition are straight forward to setup as <a href="http://kernelnewbies.org/Ext4">ext4</a> (still one of the most stables and speedy fs for linux) but the last one, <code>/usr/portage</code> is somewhat special in it's contents: mostly lots of small files (less than 1 <a href="https://en.wikipedia.org/wiki/Kibibyte">Kib</a>).</p></p>
<p>My guess is that for this kind of usage it's a waste of space to only handle blocks of 4 KiB. At the expense of needing more space to have more <a href="https://en.wikipedia.org/wiki/Inode">inodes</a> (the entries in the tree that describe the files that are on the disk) I rather have the filesystem managing more or less one block for each file.</p>
<p>Speed is also important in this partitions. The fastest it's possible to transverse all of its information the better for when using the portage tools. My gut feeling is that the easiest way to speed up reading information from the disk is to reduce the amount of data needed to be read. Therefore having small inodes is also a plus.</p>
<p>But enough chat chat. What options did I actually used when creating the filesystem for the portage tree?
Well, here they are with a small description:</p>
<p><code>-b 1024</code>
: use 1024 bytes per block</p>
<p><code>-i 1024</code>
: one inode for each 1024 bytes of space, so one inode for each block so that in the limit we could address all the blocks in the filesystem</p>
<p><code>-I 128</code>
: set the size of each inode to its minimum, meaning that there won't be any of the fancy stuff like extended attributes (I don't need them for the portage tree)</p>
<p><code>-L PORTAGE</code>
: just labeling the filesystem</p>