draft0 - a shared blog by just some people

Menu

Entries tagged 'cat:Projects' (Page 1)

PIN Changer

I had to change the default PINs of over 200 SIM cards once. And such a situation could arise again. So I've built a PIN changer in which I just have to insert the card, wait a few seconds and it's done.

The Card Slots

SIM cards in their natural form factor aren't as fiddly to handle as they are in the form factors most people know, which is Mini SIM, Micro SIM and Nano SIM. Classical SIM cards are the same size as other smart cards. I found a card slot with an end switch on eBay. I like it when I find industry grade parts for cheap on eBay as part of some remaining stock. Additionally I've used a slide-in mini SIM slot and another, separate end switch from my parts collection in case I to change the PINs of smaller cards.

The Baseband Processor

Other parts that I've used is an Arduino Nano sized Arduino Nano nearly-clone and an A6 modem module. There are many similar modem modules designed around different but similar ICs. Many of them are cheap and widely used for DIY IoT projects. So example code for the Arduino and other help can be easily found on the web. I don't know why I went with a module with an A6. But it works fine and there are an Arduino library for it as well as cheap modem modules with it available.

(tba:voltage supply)

The Controller

Yes, Arduino may be kind of the noob go-to board and could look up how to use microcontrollers on their own finally and even if I don't want to I could finally start to use ESP32s like everybody else. But I know Arduinos and by now I'm familiar with it and it works, so, whatever. Arduino Nano is kind of my go-to form factor now because they have integraded USB, are Uno compatible and small. Unless I need more or something very specific I use Arduino Nano almost-clones with USB-C port.

The Code

The code is a real mess. It had been a long time since I had written any even halfway serious C. It may have been the first time, actually. The sketch surely is very easily improved by somebody who knows what they are doing. I intended to improve it myself. But the project is currently abandoned and The code is doing what it should in a way I initially had in mind as the goal. But I'll leave the mess of the comments in for the case that somebody wants to make out what I was thinking.


// Funktionen umschreiben: Beim Empfangen wird erwartet: 1. der AT-Command zurück, 2. eine Antwort, 3. OK oder ein ERROR.
//                         Daher sollte abgefragt werden, bis entweder OK oder ERROR kam oder 20/50/9001(?) Abfragen lang weder OK noch ERROR an kam.
//                         Die Antwort in Variable speichern? Naja, String zurückgeben halt.
//                         Antworten, die mit "^" anfangen brauchen nicht gehandlet zu werden, da keine Kommandos, die mit AT^ beginnen gesendet werden.


#include  

SoftwareSerial A6MODULE(6,7);
int intled = 13; // Internal LED
int successled = 8; // Green LED
int failled = 9; // Red LED
int wrongpinled = 10; // Orange LED

boolean debug=true;

//String commands[5] = { "AT", "AT+CPINC2", "AT+CPIN?", "AT+CLCK=\"SC\",2", "AT+CPIN=\"3010\"" };
//int command = 0;

const byte maxmsglength = 32;
char received[maxmsglength];
boolean newData = false;
String response = "";

int i=0;

/*
To check/do:
1  AT: OK?
2  is PUK required - abort
3  are less than 3 PIN attempts left? - abort AT^CPINC=?
4  is PIN disabled
       5  enable it: 0000
6  is PIN enabled
       7  unlock
8  is card unlocked
       9  change PIN: 1996
              was PIN wrong - report and leave it
*/


void setup() {
  pinMode(intled, OUTPUT);
  pinMode(successled, OUTPUT);
  pinMode(failled, OUTPUT);
  pinMode(wrongpinled, OUTPUT);
  // All LEDs turn on at the beginning and stay on during the wait period at the beginning, then turn off before communication with the A6 module starts.
  digitalWrite(intled, HIGH);
  digitalWrite(successled, HIGH);
  digitalWrite(failled, HIGH);
  digitalWrite(wrongpinled, HIGH);
  Serial.begin(9600);
  delay(500);
  digitalWrite(intled, LOW);
  digitalWrite(successled, LOW);
  digitalWrite(failled, LOW);
  digitalWrite(wrongpinled, LOW);
  A6MODULE.begin(9600);
  delay(500);
  digitalWrite(intled, LOW);
}

void loop() {
  sendtoa6("AT");
  if(getfroma6("OK")) {
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PUK")) { fail; }         // If the required password is PUK, abort.
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PIN2")) { fail; }        // If the required password is PIN2, abort.
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PUK2")) { fail; }        // If the required password is PUK2, abort.

// WAIT FOR SWITCH RELEASE FIRST
    sendtoa6("AT+CPIN?");
    getfroma6("+CPIN:SIM PIN");                      // The last non-empty response will be stored in the global response variable. Problem with this: If the A6 module sends an unsolicitated message before the response to the CPIN command, nothing gets done and the card needs to be re-inserted again.
    if(strcmp(response, "OK") == 0) {                // If already no PIN is required

//      sendtoa6("AT+CLCK=\"SC\",2"); // Ist PIN-Abfrage eingeschaltet? Oder ist es "SC"?
//      if(getfroma6(???)) { PIN-Abfrage einschalten mit 0000; }

      // ENABLE PIN HERE
      d("I don't know how to enable the PIN.");

    }
    if(response = "+CPIN:SIM PIN") {                 // If the required password is PIN, continue.
      sendtoa6("AT^CPINC=?");
      if(getfroma6("^CPINC: 3")) {                   // If not exactly 3 attempts are left, abort. (should be larger than or equal to 3, shouldn't it?)
        sendtoa6("AT+CPIN=\"0000\"");                // Freischalten mit 0000
        delay(50);
        if(!getfroma6("OK")) { fail; }               // If that was not the right password, abort.
        sendtoa6("AT+CPWD=\"SC\",\"0000\",\"1996\"");// PIN ändern
        delay(500);
        if(getfroma6("OK")) { Serial.println("Looking good."); }

//      sendtoa6("AT+CMGD=0,4"); // Should delete all SMS
//      if(getfroma6(???)) { ; }
        d("I don't know how to delete SMS.");

        sendtoa6("AT+CPIN?");
        if(getfroma6("+CPIN:READY")) {
          
          // TURN OFF A& MODULE FOR SAFE CARD REMOVAL

          digitalWrite(successled, HIGH);
        }
      } else {                                         // If not exactly 3 times left
        fail;
      }
      delay(1000);
    }
  }
}

void fail() {
  digitalWrite(failled, HIGH);               // Turn red fail LED on and ...
  d("Something failed! Ending programme.");
  while(1);                                  // ... don't do anything anymore.
}

void wrongpin() {
  digitalWrite(wrongpinled, HIGH);           // Turn yellow LED on and ...
  d("Wrong PIN! Ending programme.");
  while(1);                                  // ... don't do anything anymore.
}

//boolean getfroma6(char str[32], char str1[32], char str1[32], char str1[32], char str1[32], char str1[32]) { // Returns true if the passed (expected) message was received, false if anything else was received.
boolean getfroma6(char str[32]) { // Returns true if the passed (expected) message was received, false if anything else was received.
  boolean asexpected = false;
    for (i = 1; i < 9; ++i) {
    receivelinefroma6();
//    d("d1 "+response);

    if(received[0] == '\0') {                            // If the received message is empty
      continue;
    } else {
      response = received;
    }
    if(strstr(received, "ERROR") != NULL) {              // If the received message contains "ERROR"
      d("Received an error: "+response);
      fail;
    }
    if(received[0] == '+') {                             // If the received message starts with a "+" sign
      d("Reseived response: "+response);
    } else {
      if(strstr(received, "OK") != NULL) {               // If the received message is "OK"
        d("Received OK.");
        asexpected = true;                               // Also treat OK like the expected message. No unexpected OK should ever be sent from the A6 module. So this is fine. No, it is, really.
//        if(asexpected) { return true; }
      } else {
        if(strncmp(received,"AT",2) == 0) {              // If the received message starts with "AT"
          d("Received AT: "+response);
        } else {                                         // For any other received message
          d("Received: "+response);
        }
      }
    }
    newData = false;
    if(strstr(received, str) != NULL) {
      d("Received expected message: "+response);
      asexpected = true;
    }
  }
  if(asexpected) { return true; }
  return false;
}


//void handleresponse() {
//  response = received;
//  if(received[0] == '+') {
//    d("               Response: "+response);
//  } else {
//    if(strstr(received, "OK") != NULL) {
//      d("               It's okay.");
//    } else {
//      if(strncmp(received,"AT",2) == 0) {
//        d("               I've sent: "+response);
//      }
//    }
//  }
//  newData = false;
//}



void receivelinefroma6() {
  delay(80);
  static byte counter = 0;
  char rc;
  
  received[0] = '\0';

  while (A6MODULE.available() > 0 && newData == false) {
    rc = A6MODULE.read();

    if (rc != '\n') {
      received[counter] = rc;
      counter++;
      if (counter >= maxmsglength) {
          counter = maxmsglength - 1;
      }
    }
    else {
      received[counter] = '\0'; // terminate the string
      counter = 0;
      newData = true;
    }
  }
}

void sendtoa6(String command) {
//  Serial.println(command);
  A6MODULE.println(command);
  d("Sent: "+command);
}

void d(String line) {
  if (debug == true) { Serial.println(line); }
}

(tba:connections,assembly,photos?)

Comment via email

SBWG 0.10.7

So many ideas, so little time. So many todos, no little time. So little time, so much rest of my life. But the todo list has actually started to become a little shorter for the first time in a long time. As with the previous entries about my progress with SBWG, I'm not going to address or list everything that I've improved or added. But I feel like it's time for another update.

File attachments pretty much work as intended now. Text files still can't be file attachments to entries, because text files are entries. But I think that will stay as it is for now because I don't need to attach text files to entries and if I would, I could gzip, bzip or 7z them. I mainly wanted file attachments for entry images and podcast support. Both works as well as I need it to. But some things could still be improved in the future.

Referencing tags are improved, their notes look better and there is now a way to mark an entry as an update of an other entry, giving both entries a note that tells the reader about the newer/older version. And since this works now, I dropped the idea to introduce any form of content revisioning. An entry revision system would be over the top for a feature that I hardly use. This could still be implemented just with hooks.

SBWG was relying on a feature of ImageMagick's mogrify that turned out to be a bug and was fixed. So I had to switch to convert with worse tolerance/support for broken image files. That's annoying, but I'm over it now. I mean, I wasn't complaining. Those tools are amazing and I'm so glad that they are free.

There were a number of other bugs that I've fixed. Some of them old and overlooked. But most of them were introduced not long ago. Well, I guess that's what happens if you don't do proper tests every time. SBWG is starting to become a feature-rich tool and every new feature that gets implemented could cause trouble with other features or combinations of features. In early versions of SBWG I didn't always test everything I changed. I now do, and I now also try to think of some related things the change could have effected unintentionally and test those basically. But proper tests of all sorts of scenarios every time I publish a new version? That would be professional but not worth the effort in this case. If I'm not using a feature, it doesn't matter much if I break it. I think I'll probably do more extensive tests every now and then.

Entries can now have comment links. If there is an email address available for an author, all their entries get an email link below them to allow readers to reply to them. For a public blog it might be best to use throwaway addresses that can quickly be replaced if they start to receive spam. I'd have some ideas to improve the email composed by this comment feature. But they would have to become HTML emails. So I forgot about them again.

New tag types, expanded README file, improved navigation by adding parent links and things like this, improved CSS a bit here and there, very small improvements to the code style. I didn't make any relevant improvements or corrections in terms of code quality. I think this is where SBWG could improve most. It's not a professional thing, so I declare myself fine with the way it is (which isn't that bad compared to some other software, I think) but I could learn a lot there.

Command line options can now be grouped to make typing commands easier. Long options (starting with --) didn't change (I may make those things POSIX-like, too, some other time) but short options (starting with a -) now don't need to be separated and all have their own dash. So sbwg -v -p -e blub can now be shortened to sbwg -vpe blub. And if you want to run the command again but with added verbosity, you can add the second -v wherever you want.

There is now a waiting animation that starts to play if there is no update on stdout for three seconds. Both the animation frames and the time delay can be changed. The animation is only enabled if SBWG runs with any verbosity. Then it is enabled by default. It will be disabled by default in future versions though, I think, because it's such an unnecessary messing around with terminal output. It does work pretty well though. I thought it may be harder to make it work as smoothly as it does now (which isn't 100% perfect, but pretty good).

The new default style set 'elth' is a working hybrid with different styles for small phone screens and larger screens of phablets, tablets and laptops. It's composed of several CSS files, which should make it easier to create a custom style now, since just the basics (without any colors and fancy embolishment and without any tag-specific stuff) can be copied for a template, or just the mobile style can be copied and used to create a new mobile style. Style sets can now also include files whose name don't end in '.css'. That means they can contain any file type. So JS or background images or whatever will be copied as part of the style set and placed in the 'styles/' directory of the generated web site.

More things are now cached that may be used more than once in one run of SBWG. Permament caches (option -C/--cache) can now be enabled for only parts of the web site. I still need to decide how permanent caches should be managed. The way I had/have it in mind does not seem very intuitive. It is to me because I know how it's built. But... well, maybe it's enough that I think that it makes sense. Maybe I should just start using it and if it works well, write it down in the README file.

Added new hooks, gave the helper scripts makeovers, made how HTML footers are made logical, sorted much of the remaining todos. Overall: progress.

So, I've published SBWG version 0.10.7 now. I wish I'll find the time to create and publish new versions more frequently again in the near future. But I doubt I will.

Comment via email

Not turning my old Laptop into a cool-looking, silent HTPC

When my old laptop was finally broken enough that I bought a new one, I disassembled it and reluctantly put its components next to a Pentium II board and an awesome old SCSI hardware RAID controller into a drawer. The display is still good, and so is its motherboard and IO boards. Still a working peace of tech, but without a working battery, solid case and fully functioning keyboard it's of no use to me. Unless I had a use case for another computer that's neither good for travelling nor powerful enough to justify being a stationary PC, this thing would rot in this drawer. But I had an idea for a case that would give it a new purpose as a media station. One mSATA system SSD, three 2.5" media HDDs (or SSDs), a wooden enclosure, six large copper heat sinks that would protrude the left wall to allow for passive cooling of the CPU and GPU. The picture in my mind is really nice, it has a 4th gen mobile i7 CPU, a decent GPU, a good enough DAC, all relevant video outputs, I have the parts, drive bays, heat sinks, heat pipes, wood, ... But, it was apparently never important enough to me to get started. I doubt I'd use an HTPC much. And so it is still rotting in that drawer.

Does anybody want that PC? I'd build it.

Now my new laptop has started to degrade. Battery, keyboard, touch pad. all the parts that wouldn't be required to turn it into a stationary HTPC. It lacks the four SATA ports. But it has a fast NVME and a fast enough USB for more than four SATA ports, needs less power, the mainboard is much smaller. So it would also be a good candidate for such a PC. But I still don't need it except as a travel notebook. Sooo, I guess that's the end of this PC case idea.

Comment via email

SBWG 0.10.2

I know, my entries on SBWG are too chaotic to follow what's happening with the project in detail. But who cares, really? I'll just continue to write broadly about the progress that I make with the script.

So, after re-deciding that I'm not going to make a version 1.0 because it's just not the kind of software that can ever claim for itself the thoroughness that I myself would expect from a piece of software that's labeled v1.0, I finished the testing and bug squashing that I, for a short while, had intended to lead to version 1.0 to my satisfaction, and started implementing new features again. Great, now I can make the script take longer to finish again. :)

During my testing and bug squashing spree I already implemented a routine that I call the pathreducer. If enabled through command line option, it reduces paths to conform with the file and directory name restrictions of most file systems out there. I took this a bit far and it can now even conform to filenames of the 6.3 format, if the user wishes. I have ideas to improve this further, including making it automatically conform to the exact limitations of individual file systems instead of stripping and replacing characters that aren't allowed in file names in many file systems whether it would have been necessary in the present case or not. But that's for another time. For now it works well for what it is supposed to be: a workaround for too long and otherwise invalid file names that would be generated by the script when naming files after categories, topics and other tags, as well as a solution to problems that might occur if SBWG would be used to generate a web site onto a file system that has more restrictions than those commonly used on Unix-like systems.

As part of the improvements made during the no-new-feature time, I fixed many bugs that I didn't expect at all when I started (which is why I prohibited myself from implementing new features in the first place), learned a lot about Bash again, and made some small improvements that I wouldn't call new features, but are still neat to have. One of them makes tag lines more intuitive. Forbidden characters are removed or replaced and leading and trailing whitespaces are removed, leading coincidentally to the fact that you can use a space after the tag type (e.g. 'cat: Example' instead of 'cat:Example') or not and the tag will be recognised as being identical in both cases. I'm mentioning this because the first person to type a SBWG tag line besides me typed a space intuitively and I had to tell them that that's not how that works. Well, now it is, if you prefer it. Maybe case-insensitivity will be next. I'm considering it. Wouldn't be hard to add. I'm not sure if I want it, though.

Another thing that improved drastically in that time is the time it takes to generate huge web sites. SBWG never had problems with giant files, even entry files with gigabytes of text, because the content body isn't parsed. But with thousands of entries or thousands of tags in one entry, generation time would previously go into hundreds of years (theoretically/calculated, of course). By improving some routines and caching some parts during the generation process I got the generation time of an example torture web site that I created, from over 300 years down to just short of 72 hours on my Core-i5 ultrabook onto an NVMe drive. Any reasonably sized web site even with a relatively large weblog is completely generated much quicker. So it would be possible to e.g. let SBWG re-generate a web site automatically multiple times a day without problems. I have other ideas that will speed up the generation process significantly, namely parallel processing and skipping of existing unchanged content. But both those features are not mature enough, yet. I will probably attend to them again when I have packed SBWG with so many features that slow down the process compared to the current version that I feel more parallelisation and more drastic caching will be necessary.

So, the first new feature on the list were entry pictures. A way to bind pictures, image files, to entries without having to include <img> tags. This feature now already developed into a general way to attach files to entries. Images are resized, embedded and linked to their originals, audio files are embedded as an audio player if the browser supports this, and other files are linked to allow visitors to download them. Video files will also be embedded as a player in future version. The file attachments feature is not done, yet. Enclosures in the RSS feed, presentation of the download links and many small things have to be improved. But the initial idea of entry pictures already works. There is a naming convention that makes a file an attachment to a certain entry. (The same naming convention as is used for style sheet sets.) See the README of version 0.10.2 or above for detailed information.

The next feature on the list is comment links below blog entries. Simple mailto links to let readers e-mail comments on entries to the author. I will probably start to work on multiple features at the same time again, as I did before.

So, I've published SBWG version 0.10.2 now. I hope I'll find the time to create and publish new versions more frequently again in the near future. But I doubt I will.

Comment via email

All Those Bike Attachements

I think bicycle accessaries are of these things that developed over time past the point where it would have been sensible to rethink how things are done. I think it would make sense the rethink how all those bike attachments are integrated.

A bicycle without any attachments is already nice. It can be used without them and none of the things are necessary all the time. So it can make sense to have them detachable. But probably most people use their bike mainly for one thing - transportation, mountain-biking, sports - and need the same combination of attachments most times the bike is used. (Minus the lights when it's not dark.) I think it's rather peculiar that people buy, attach and use all these extras without questioning the crowded state of their bicycle frame. Let me list the things that I find useful myself.

  • Bell (*)
  • Front light (*)
  • Back light (*)
  • Phone/GPS holder
  • Phone charger
  • Action camera
  • Remote for indicators (in helmet or jacket/shirt)
  • Horn

(*): required by law when driving on the street

So, in this configuration, when you get back from riding your bike, you have to turn off the front light, turn off the back light, turn off the helmet indicator, turn off the indicator remote, check the charge of the front light, the back light, the phone charger, the GPS, the helmet, the helmet control, the horn and the action cam, detach the things that need charging, bring them in the house and plug them in for charging one after another. You may also have an electronic lock, additional front lights, a breaking light, the remote control for the breaking light and a headlamp. Half of them still have a micro USB port and none of the batteries last for dozens of rides (unless you ride mostly when it isn't dark). Sure, that comes with the benefit of having these devices, people seem to think. But they don't do all of this when they get out of their car. Because it wouldn't make sense to have a separate battery for every electronic device in a modern car. Neither does it make sense on a bike.

It would be nice to have a central battery for everything on my bike and a single on/off switch. But I don't use it enough to see me investing in building it myself. Because if I'd do it, I'd want it to be safe and secure in any weather, look good and not be too clunky or heavy. So I'd have to invest time.

Comment via email

SBWG 1.0.0 Delayed To Next Life

I've decided to not publish version 1.0.0 of SBWG.

Very early on I devided against a versioning approach that allowed me to stay below version 1.0 for a very long time by only incrementing the version number by 0.0.1 for important changes but rather chose to increment the version and publish a new version whenever. I stick to this approach because it allowes me to express felt overall progress in the version number. But now that the goals that I at some point set for version 1.0.0 are nearly reached (they pretty much are), this leaves me in a spot where publishing version 1.0.0 would be the next thing to do despite the fact that it's not actually really absolutely perfect, yet. Absolute perfection isn't really my approach. It's still more like a learning project.

So to avoid having published a version that looks perfect on the label but isn't inside, I will skip this version number. Don't get me wrong. My goals are. I've tested it more than I thought I would and thought of potential problems and fixed bugs that I'd argue are not something you'd expect most shell scripters to catch. It definitely works reliably for what it is intended and at least works well for much more than I thought it would in the first few months of starting this project.

So, there is no link in this entry. No new version published today. The next version will be 1.x.x something.

Alright. Now that that's done, I can start to implement new features again next year.

Comment via email

backup.sh

This entry is an update of the entry 'Backup Shell Script'.

I've updated/improved my backup script again.

Download


#!/bin/bash
#2021-11-03

budir="/mnt/backup"
num=8
list="/root/backup-list"
hostname=$(</etc/hostname)
srcdir="$1"
#NAME=${SRCDIR//\//--}
name="$2"

function usage () {
  echo "This script requires either one or two arguments."
  echo "Usage:"
  echo "$0 JOBLIST"
  echo "$0 SOURCE_DIRECTORY NAME"
  echo
  echo "JOBLIST File that contains one backup job per line in the format SOURCE_DIRECTORY NAME"
  echo "SOURCE_DIRECTORY Directory that should be backed up. Please, no trailing slash."
  echo "NAME A string that is used to name the backup in the destination."
  exit 1
}

function single() {
  if grep -qs "$budir " /proc/mounts
  then
    fullname="$budir/$hostname/$name"
    printf "\n"
    printf "Attempting backup of source directory '%s' to '%s'.\n" "$srcdir" "$fullname"
    printf "Number of differential backups to keep: %s\n" "$num"
    printf "Removing oldest backup... "
#    rm -rf "$fullname.$num"
    rsync --archive --delete "$budir/empty/" "$fullname.$num/"		# This is quicker than rm.
    printf "Done.\n"
    for ((i=$num; i>=2; i--)); do
      printf "Renaming '%s' to '%s' ... " "$fullname.$((i-1))" "$fullname.$i"
      mv "$fullname.$((i-1))" "$fullname.$i"
      printf "Done.\n"
    done
    printf "Duplicating last backup ('%s' to '%s')... " "$fullname.0" "$fullname.1"
    [[ -d $fullname.1 ]] && exit 1							# This directory should not exist at that point.
    cp -al "$fullname.0" "$fullname.1"
#    rsync --archive --acls --xattrs --hard-links "$fullname.0/" "$fullname.1/"
    printf "Done.\n"
    now=$(date)
    printf "\n\n\n" >> "$fullname.log"
    printf "STARTING INCREMENTAL BACKUP AT %s\n" "$now" >> "$fullname.log"
    printf "Starting new incremental backy uppy at '%s.0'..." "$fullname"
    if rsync --exclude ".cache" --archive --no-links --delete "$srcdir/" "$fullname.0/" 2>&1 | tee -a "$fullname.log"
    then
      printf " Done.\n"
      printf "Done.\n" >> "$fullname.log"
    else
      printf " Failed.\n"
      printf "Failed.\n" >> "$fullname.log"
    fi
    now=$(date)
    printf "\nIf this line is here the script finished (with or without errors) at %s\n" "$now" >> "$fullname.log"
  else
    printf "'%s' is not mounted. Aborting." "$budir"
    exit 1
  fi
}

function fromlist() {
  while read job; do
    if [ -n "$job" ]; then
      name="${job#* }"
      srcdir="${job% *}"
      single
    fi
  done < "$list"
}

mkdir "$budir/empty"									# Empty directory for a quicker method to delete a large directory.

case $# in
0)
  mount "$budir"
  printf "Reading backup jobs from list. Defaulting to %s.\n" "$list"
  fromlist
  ;;
1)
  mount "$budir"
  list="$1"
  printf "Reading backup jobs from %s." "$list"
  fromlist
  ;;
2)
  mount "$budir"
  single
  ;;
3)
#  I can't decide whether to make the third argument num or budir. I don't need it anyway.
#  num=$3
#  budir=$3
#  mount $budir
#  single
  usage
  ;;
*)
  usage
  ;;
esac

exit 0
Comment via email

SBWG - The Pathshortener And Other Recent Changes

I made it my goal to harden SBWG before I start to implement new features. Before I call the next version of this project 1.0 I want to make sure that unexpected input from the command line or from source files, absurd numbers of absurdly long tags and content items, stupidly weird filenames or random binary data as tag values as well as purposfully created traps in the various places where input is processed are handled well, meaning that nothing fails unless there is no sensible way around it, and if something fails, that nothing breaks. Data should be filtered carefully, errors should be handled well and whereever possible data should be made processible if it was supplied in an unprocessible form to reduce the chances of errors. On top of that I wanted to make sure that the script did its job in a reasonable amount of time considering the circumstances. I mean, it will never be very fast. Bash is just not the right language for that. But there certainly were some repetitive tasks that could be improved. Fot the latter I created a simple caching functionality that will probably be extended in the future. I managed to reduce the (calculated/estimated) generation time of my biggest test web site from almost 300 years to a few days. Actual web sites will of course not take that long to generate, even on a slow machine. A huge web site will maybe take up to one day to generate completely, even without the new options that keep the script from re-generating existing unchanged parts of the web site. But before sombody will try to create such a big web site with SBWG I will probably have improved speed further. And even then it's a worst-case time.

As part of the aforementioned goals I have started working on last new feature before version 1.0. I call it the pathreducer. Since many of the files created by SBWG are named after the tags they represent, they can become quite long and contain almost any printible character, including multi-byte unicode characters or characters of character sets I haven't even heard of. I definitely don't want to restrict more than I already have what characters and how many of them tag values can contain. Especially filesystems used by operating systems from Microsoft are relatively restrictive in maximum allowed directory, path and filename length and allowed characters. By default the pathreducer is not used. But if enabled via command line option or in a web site's settings file, it will filter directory and filenames and shorten them to a user-defined maximum length. If the pathreducer decides to change a path elements it also adds a 6-character hash value to make shortened or otherwise reduced path elements as good as unique.

That works well for now and even can create 8.3 or 6.3 filenames for old DOS filesystems. But the result is not very nice because it isn't aware of what filesystem it is going to write a file to. To be save it removes more characters than it would have to for ext and NTFS filesystems. In the future I may extend the pathreducer to detect the filesystem at least of the root of the output directory automatically and decide how exactly path elements should be reduced according to the actual limitations of the present filesystem. Than it may even be enabled by default, even though it can increase the generation time quite a bit.

There are still some tests that I want to do and I will probably find some more things that I want to fix before version 1.0. But I see light.

Comment via email
Mastodon