Software

by

Introducing FileStruct (for Python)

FileStruct is a lightweight and fast file-cache / file-server designed for web-applications.  It solves the problems of “where do I save all of those uploads” that has been encountered time and time again.  FileStruct uses the local filesystem, but in a sensible way (keeping permissions sane), and with the ability to secure it to a reasonable level.

https://github.com/appcove/FileStruct/

Here is a simple example of taking an image upload, resizing, and saving it:

with client.TempDir() as TempDir:
   open(TempDir.FilePath('upload.jpg'), 'wb').write(mydata)
   TempDir.ResizeImage('upload.jpg', 'resize.jpg', '100x100')
   hash1 = TempDir.Save('upload.jpg')
   hash2 = TempDir.Save('resize.jpg')

Design Goals

Immutable Files

FileStruct is designed to work with files represented by the SHA-1 hash of their contents. This means that all files in FileStruct are immutable.

High Performance

FileStruct is designed as a local repository of file data accessable (read/write) by an application or web application. All operations are local I/O operations and therefore, very fast.

Where possible, streaming hash functions are used to prevent iterating over a file twice.

Direct serving from Nginx

FileStruct is designed so that Nginx can serve files directly from it’s Data directory using an X-Accel-Redirect header. For more information on this Nginx configuration directive, see http://wiki.nginx.org/XSendfile

Assuming that nginx runs under nginx user and file database is owned by the fileserver group, nginx needs to be in thefileserver group to serve files:

# usermod -a -G fileserver nginx

Secure

FileStruct is designed to be as secure as your hosting configuration. Where possible, a dedicated user should be allocated to read/write to FileStruct, and the database directory restricted to this user.

Simple

FileStruct is designed to be incredibly simple to use.

File Manipulaion

FileStruct is designed to simplify common operations on files, especially uploaded files. Image resizing for thumbnails is supported.

Temporary File Management

FileStruct is designed to simplify the use of Temp Files in an application. The API supports creation of a temporary directory, placing files in it, Ingesting files into FileStruct, and deleting the directory when completed (or retaining it in the event of an error)

Garbage Collection

FileStruct is designed to retain files until garbage collection is performed. Garbage collection consists of telling FileStruct what files you are interested in keeping, and having it move the remaining files to the trash.

Backup and Sync with Rsync

FileStruct is designed to work seamlessly with rsync for backups and restores.

Atomic operations

At the point a file is inserted or removed from FileStruct, it is a filesystem move operation. This means that under no circumstances will a file exist in FileStruct that has contents that do not match the name of the file.

No MetaData

FileStruct is not designed to store MetaData. It is designed to store file content. There may be several “files” which refer to the same content. empty.log, empty.txt, and empty.ini may all refer to the empty fileData/da/39/da39a3ee5e6b4b0d3255bfef95601890afd80709. However, this file will be retained as long as any aspect of the application still uses it.

Automatic De-Duplication

Because file content is stored in files with the hash of the content, automatic file-level de-duplication occurs. When a file is pushed to FileStruct that already exists, there is no need to write it again.

This carries the distinct benifit of being able to use the same FileStruct database across multiple projects if desired, because the content of file Data/da/39/da39a3ee5e6b4b0d3255bfef95601890afd80709 is always the same, regardless of the application that placed it there.

Note: In the event that multiple instances or applications use the same database, the garbage collection routine MUST take all references to a given hash into account, across all applications that use the database. Otherwise, it would be easy to delete data that should be retained.

by

How to Generate a SSH Keypair (public/private) on Windows

Have you ever been asked to generate an SSH keypair in order to gain access to a server, github, or an sftp site?

Here is how on windows.

First, download puttygen.exe from here:

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

 

Second, run puttygen.exe and follow these instructions:

(except, put your name instead of Sharon)

(On step 8, copy and paste this and send it to whomever requested it)

puttygen instructions

by

Simulating ENUM in PostgreSQL using CHECK expression

PostgreSQL is a very powerful database. One of the things that seems missing when moving from MySQL is the ability to simply create an enumeration. ENUM is nice when you have a programmatically-semantic set of values for a field.

In PostgreSQL, you have several choices. But one simple one is to create a Check expression, like follows. Skip the IS NULL part if you don’t want the field nullable.

ALTER TABLE 
   "public"."CallCenter_Transfer"
ADD CONSTRAINT 
   "CallCenter_Transfer_TransferStatus"
   CHECK (
      "TransferStatus" IS NULL 
       OR 
      "TransferStatus" = ANY(ARRAY['TransferComplete', 'TransferFailedNoAnswer', 'TransferFailedProspectLost'])
      )
by

git checkout -b –no-track

Ever want to checkout a new git branch from another branch without setting up tracking?

Here is the longhand way:

git checkout old-branch
git branch new-branch
git checkout new-branch

But there is a quicker way:

git checkout -b new-branch old-branch

… which does the same thing, albiet in one command.

by

HomeSchool Software: Review, Rendering, and Printing

After several intense programming sessions, I got the HomeSchool software in a quite suitable place for us to be able to use it to produce a log and portfolio.

I must say, it’s looking pretty nice!  It is rather amazing to look back at the year, and see in detail what was going on with Eli.

One of my favorites is on page 159 (read the letter on the telephone pole):

 

Here is a screen shot of the book view:

by

HomeSchool Software: Getting Close

I’ve recently been working against a deadline this month to get our Portfolio rendered, printed, and turned into the local Altoona Area School District.

The software has many improvements from the last couple days alone.  A background page renderer runs at all times, and will re-render a book’s page if anything on it changed.  The user interface for positioning images on a page is nearly complete.  I’ve enhanced the database with storage of full size JPEG images as well as info about the image (width, height, type, size).

Here is a screen shot of the page editor:

by

HomeSchool Software: Page Rendering (Also ImageMagick Text Example)

Last couple days have had me working on the page rendering functionality of the HomeSchool software.  I’ve made some design decisions (for now) to speed up development like fixed 8.5″ x 11″ pages.  Hopefully later I will be able to go back and improve some aspects, but for now, time is of the essence.

Here is an example page render without the photographs included.  Still need a UI for selecting photographs and positioning them on the white area of the page.

At the bottom, see an ImageMagick Text Example about the python code used to render multiple text areas and color blocks on one image…

It took me a long time to figure out how to use ImageMagick to place multiple text areas on one image.  Here are some keywords for google:

ImageMagick, ImageMagick multiple text areas, ImageMagick pdf rendering, ImageMagick text.

      PrimerFont = join(App.Path, 'Static', 'Primerb.ttf')
      PrimerFontBold = join(App.Path, 'Static', 'Primer.ttf')
      RenderFile = join(TempDir, 'Render.jpg')

      # Build the full image
      subprocess.check_call((
        '/usr/bin/convert',

        # Setup the background
        '-size', '2550x3300',
        'xc:white',

        # Title Bar
        '-fill', 'grey14', '-draw', 'rectangle 0,0 2550,300',

        # Summary bar
        '-fill', 'RoyalBlue4', '-draw', 'rectangle 0,300 2550,600',

        # Description Bar
        '-fill', 'LightGrey', '-draw', 'rectangle 0,600 2550,1050',

        # Now draw the Name
        '(',
          '-density'    , '300',
          '-fill'       , 'white',
          '-background' , 'grey14',
          '-font'       , PrimerFont,
          '-size'       , '1650x150',
          '-gravity'    , 'West',
          '-pointsize'  , '20',
          'caption:'    + self.Book.Name,
        ')',
        '-gravity', 'NorthWest', '-geometry', '+150+150', '-composite',

        # Now draw the Date
        '(',
          '-density'    , '300',
          '-fill'       , 'white',
          '-background' , 'grey14',
          '-font'       , PrimerFont,
          '-size'       , '600x150',
          '-gravity'    , 'East',
          '-pointsize'  , '20',
          'caption:' + Activity.Date.strftime('%B %d, %Y'),
        ')',
        '-gravity', 'NorthEast', '-geometry', '+150+150', '-composite',

        # Now draw the Summary
        '(',
          '-density'    , '300',
          '-fill'       , 'white',
          '-background' , 'RoyalBlue4',
          '-font'       , PrimerFontBold,
          '-size'       , '2250x300',
          '-gravity'    , 'Center',
          '-pointsize'  , '20',
          'caption:'    + Activity.Summary.replace("\n", " "),
        ')',
        '-gravity', 'North', '-geometry', '+0+300', '-composite',

        # Now draw the Description
        '(',
          '-density'    , '300',
          '-fill'       , 'black',
          '-background' , 'LightGrey',
          '-font'       , PrimerFont,
          '-size'       , '2250x450',
          '-gravity'    , 'Center',
          '-pointsize'  , '14',
          'caption:'    + Activity.Description.replace("\n", "\\n"),
        ')',
        '-gravity', 'North', '-geometry', '+0+600', '-composite',

        # Now draw the Page
        '(',
          '-density'    , '300',
          '-fill'       , 'black',
          '-background' , 'white',
          '-font'       , PrimerFont,
          '-size'       , '2250x150',  #entire right half minus 150pt margin
          '-gravity'    , 'Center',
          '-pointsize'  , '14',
          'caption:'    + 'Page {0}'.format(self.PageNumber),
        ')',
        '-gravity', 'South', '-geometry', '+0+150', '-composite',

#        NameFile, '-geometry', '+0+0', '-composite',
#        DateFile, '-geometry', '+1275+0', '-composite',
#        SummaryFile, '-geometry', '+0+150', '-composite',
#        DescriptionFile, '-geometry', '+0+450', '-composite',

        RenderFile,
        ))
by

Home School Software Update: Portfolio Time

It’s that time of year again… Time to get the log and portfolio to the Altoona Area School District.  One problem — the software isn’t quite ready!

However, I’ve been making daily progress on getting the “output” working.  Here is how it is structured in this initial version:

  • You can create a book, “Eli Daniel Garber 2010-2011 Portfolio” for example.
  • Activities are added to the book, one activity per page.
  • The activity summary and description, date and pictures come along.
  • You can move the pictures around, crop, and choose the exact ones you want.
  • Pages in the book can be sorted any way you want.
  • Once the book is “ready”, you can output a print-quality PDF.  This PDF could be printed or sent to a printer to turn into a hardback book.
So far, we have recorded about 500 activities this year covering nearly 200 days — and we are still catching up on things that didn’t get entered.  One of the neat aspects of this project is to be able to really see what we’ve been up to all year.
Due to time constraints, I’m just focusing on getting good output.  Bells and whistles (chapters, cover pages, extra documents, table of contents, etc…) will have to come later.
Also, I added extensive filtering to the activity list, to help make it faster and more manageable.  Here are a couple of screen shots: