CiprianDorinCraciun/Notes/Ideas/WebCat

web-cat

Description

A tool similar in principle with netcat (actually socat), that basically acts as a byte conveyor between a source stream and a sink stream, the sink being represented by HTTP-response sent as a reply for an HTTP-request received through a web-server.

Use-cases

• continuously exporting logs over HTTP;
• testing web-clients that expect a streaming HTTP server;
• a simple way to distribute one-way, fire-and-forget events to multiple clients;

Required skills

• software engineering	beginner
  programming	medium
  web-technologies	beginner
  operating systems	beginner

Basic features

• it always conveys a record, where the meaning of record depends on the encoding of the source (input) and sink (output) streams; but in general they could be:

  • an arbitrary block of data (no boundary);

  • a length-value binary packet;

  • an UTF-8 line of text;

  • a block of data, which follows a certain syntax (like a JSON object);

• the encoding of the source (input) stream can be either:

  • an UTF-8 text stream, where each record is represented by a line of text;

  • an UTF-8 text stream, but where each record is delimited by a \0 byte;

  • a length-value binary packet (see decode_packet, the 1 | 2 | 4 type);

  • a stream of chunks as presented in Chunked Encoding;

• the source (input) stream can be represented by either:
  • a static file;

  • a pipe, obtained maybe:

    • as stdin (or an arbitrary descriptor) of the process;

    • as a fifo object on the file-system;

    • as stdout of an child process;

  • as a byte stream socket;

• the sink (output) stream is always a HTTP-response with the following features:
  • either unbounded-length response;

  • either Chunked Encoding;

• there could be one or multiple in parallel sinks, each receiving the same data;

Medium features

• other source stream types (all optional):

  • Datagram socket (e.g. UDP, etc.);

  • UNIX domain sockets;

  • other Linux supported sockets (e.g. raw);

  • POSIX message queues;

• features of the HTTP-stack:

  • support for HTTP Content-Encoding;

  • support for WebSockets;

• support for limited ring-buffer for late-starters;

• support for multiple sources, and the ability for sinks to select among them (a primitive publish / subscribe implementation);

• recording and replaying a capture of records (including timing information);

Advanced features

• other source stream types (all optional):

  • other suitable protocols (maybe through cURL library);

  • ØMQ message queues;

  • AMQP message queues;

• filtering support for sources or sinks based on:
  • record prefix;

  • record regular expression;

• rate-limiting;
• support for record transformation:

  • through an external filter (e.g. sed);

• features of the HTTP-stack:

  • HTTPS;

  • HTTP Basic Authentication;

  • HTTP Digest Authentication;

Programming languages

• (in order of preference)

• Go;

• C/C++;

• NodeJS;

• Python;