summaryrefslogtreecommitdiffstats
path: root/tdeio/DESIGN
diff options
context:
space:
mode:
authorTimothy Pearson <kb9vqf@pearsoncomputing.net>2013-01-27 01:04:16 -0600
committerTimothy Pearson <kb9vqf@pearsoncomputing.net>2013-01-27 01:04:16 -0600
commit5159cd2beb2e87806a5b54e9991b7895285c9d3e (patch)
tree9b70e8be47a390f8f4d56ead812ab0c9dad88709 /tdeio/DESIGN
parentc17cb900dcf52b8bd6dc300d4f103392900ec2b4 (diff)
downloadtdelibs-5159cd2beb2e87806a5b54e9991b7895285c9d3e.tar.gz
tdelibs-5159cd2beb2e87806a5b54e9991b7895285c9d3e.zip
Rename a number of libraries and executables to avoid conflicts with KDE4
Diffstat (limited to 'tdeio/DESIGN')
-rw-r--r--tdeio/DESIGN272
1 files changed, 272 insertions, 0 deletions
diff --git a/tdeio/DESIGN b/tdeio/DESIGN
new file mode 100644
index 000000000..22307c5c0
--- /dev/null
+++ b/tdeio/DESIGN
@@ -0,0 +1,272 @@
+DESIGN:
+=======
+
+libtdeio uses tdeioslaves (separate processes) that handle a given protocol.
+Launching those slaves is taken care of by the tdeinit/tdelauncher tandem,
+which are notified by DCOP.
+
+Connection is the most low-level class, the one that encapsulates the pipe.
+
+SlaveInterface is the main class for transferring anything to the slave
+and Slave, which inherits SlaveInterface, is the sub class that Job should handle.
+
+A slave inherits SlaveBase, which is the other half of SlaveInterface.
+
+The scheduling is supposed to be on a two level basis. One is in the daemon
+and one is in the application. The daemon one (as opposite to the holy one? :)
+will determine how many slaves are ok for this app to be opened and it will
+also assign tasks to actually existing slaves.
+The application will still have some kind of a scheduler, but it should be
+a lot simpler as it doesn't have to decide anything besides which
+task goes to which pool of slaves (related to the protocol/host/user/port)
+and move tasks around.
+Currently a design study to name it cool is in scheduler.cpp but in the
+application side. This is just to test other things like recursive jobs
+and signals/slots within SlaveInterface. If someone feels brave, the scheduler
+is yours!
+On a second thought: at the daemon side there is no real scheduler, but a
+pool of slaves. So what we need is some kind of load calculation of the
+scheduler in the application and load balancing in the daemon.
+
+A third thought: Maybe the daemon can just take care of a number of 'unused'
+slaves. When an application needs a slave, it can request it from the daemon.
+The application will get one, either from the pool of unused slaves,
+or a new one will be created. This keeps things simple at the daemon level.
+It is up to the application to give the slaves back to the daemon.
+The scheduler in the application must take care not to request too many
+slaves and could implement priorities.
+
+Thought on usage:
+* Typically a single slave-type is used exclusively in one application. E.g.
+http slaves are used in a web-browser. POP3 slaves used in a mail program.
+
+* Sometimes a single program can have multiple roles. E.g. konqueror is
+both a web-browser and a file-manager. As a web-browser it primarily uses
+http-slaves as a file-manager file-slaves.
+
+* Selecting a link in konqueror: konqueror does a partial download of
+the file to check the mimetype (right??) then the application is
+started which downloads the complete file. In this case it should
+be able to pass the slave which does the partial download from konqueror
+to the application where it can do the complete download.
+
+Do we need to have a hard limit on the number of slaves/host?
+It seems so, because some protocols are about to fail if you
+have two slaves running in parralel (e.g. POP3)
+This has to be implemented in the daemon because only at daemon
+level all the slaves are known. As a consequence slaves must
+be returned to the daemon before connecting to another host.
+(Returning the slaves back to the daemon after every job is not
+strictly needed and only causes extra overhead)
+
+Instead of actually returning the slave to the daemon, it could
+be enough to ask 'recycling permission' from the daemon: the
+application asks the daemon whether it is ok to use a slave for
+another host. The daemon can then update its administration of
+which slave is connected to which host.
+
+The above does of course not apply to hostless protocols (like file).
+(They will never change host).
+
+Apart from a 'hard limit' on the number of slaves/host we can have
+a 'soft limit'. E.g. upon connection to a HTTP 1.1 server, the web-
+server tells the slave the number of parallel connections allowed.
+THe simplest solution seems to be to treat 'soft limits' the same
+as 'hard limits'. This means that the slave has to communicate the
+'soft limit' to the daemon.
+
+Jobs using multiple slaves.
+
+If a job needs multiple slaves in parallel (e.g. copying a file from
+a web-server to a ftp-server or browsing a tar-file on a ftp-site)
+we must make sure to request the daemon for all slaves together since
+otherwise there is a risk of deadlock.
+
+(If two applications both need a 'pop3' and a 'ftp' slave for a single
+job and only a single slave/host is allowed for pop3 and ftp, we must
+prevent giving the single pop3 slave to application #1 and the single
+ftp slave to application #2. Both applications will then wait till the
+end of times till they get the other slave so that they can start the
+job. (This is a quite unlikely situation, but nevertheless possible))
+
+
+File Operations:
+listRecursive is implemented as listDir and finding out if in the result
+ is a directory. If there is, another listDir job is issued. As listDir
+ is a readonly operation it fails when a directory isn't readable
+ .. but the main job goes on and discards the error, because
+bIgnoreSubJobsError is true, which is what we want (David)
+
+del is implemented as listRecursive, removing all files and removing all
+ empty directories. This basically means if one directory isn't readable
+ we don't remove it as listRecursive didn't find it. But the del will later
+ on try to remove it's parent directory and fail. But there are cases when
+ it would be possible to delete the dir in chmod the dir before. On the
+ other hand del("/") shouldn't list the whole file system and remove all
+ user owned files just to find out it can't remove everything else (this
+ basically means we have to take care of things we can remove before we try)
+
+ ... Well, rm -rf / refuses to do anything, so we should just do the same:
+ use a listRecursive with bIgnoreSubJobsError = false. If anything can't
+ be removed, we just abort. (David)
+
+ ... My concern was more that the fact we can list / doesn't mean we can
+ remove it. So we shouldn't remove everything we could list without checking
+ we can. But then the question arises how do we check whether we can remove it?
+ (Stephan)
+
+ ... I was wrong, rm -rf /, even as a user, lists everything and removes
+ everything it can (don't try this at home!). I don't think we can do
+ better, unless we add a protocol-dependent "canDelete(path)", which is
+ _really_ not easy to implement, whatever protocol. (David)
+
+
+Lib docu
+========
+
+mkdir: ...
+
+rmdir: ...
+
+chmod: ...
+
+special: ...
+
+stat: ...
+
+get is implemented as TransferJob. Clients get 'data' signals with the data.
+A data block of zero size indicates end of data (EOD)
+
+put is implemented as TransferJob. Clients have to connect to the
+'dataReq' signal. The slave will call you when it needs your data.
+
+mimetype: ...
+
+file_copy: copies a single file, either using CMD_COPY if the slave
+ supports that or get & put otherwise.
+
+file_move: moves a single file, either using CMD_RENAME if the slave
+ supports that, CMD_COPY + del otherwise, or eventually
+ get & put & del.
+
+file_delete: delete a single file.
+
+copy: copies a file or directory, recursively if the latter
+
+move: moves a file or directory, recursively if the latter
+
+del: deletes a file or directory, recursively if the latter
+
+PROGRESS DISPLAYING :
+=====================
+Taj brought up the idea of deligating all progress informations to an extern
+GUI daemon which could be provided in several implementations - examples
+are popup dialogs (most are annoyed by them, like me :) or a kicker applet
+or something completely different. This would also remove the dependency on
+libtdeui (I hope).
+Conclusion: tdeio_uiserver is this single GUI daemon, but the dependency on
+libtdeui couldn't be removed (for many reasons, including Job::showErrorDialog())
+
+A. progress handling
+---------------------
+There will be two ways how the application can display progress :
+
+1. regular apps will use NetAccess for all kio operations and will not care
+ about progress handling :
+ - NetAccess creates Job
+ - NetAccess creates JobObserver that will connect to the Job's signals and
+ pass them via dcop to the running GUI Progress Server
+
+2. apps that want to do some handling with progress dialogs like Caitoo or
+ KMail :
+ - app creates Job
+ - app creates a progress dialog : this should be a ProgressBase descendant
+ e.g. StatusProgress or custom progress dialog
+ - app calls progress->setJob( job ) in order to connect job's signals with
+ progress dialog slots
+
+B. customized progress dialogs
+-------------------------------
+ This will be similar to what we had before.
+
+ - ProgressBase class that all other dialogs will inherit.
+ will contain an initialization method setJob( TDEIO::Job*) for apps of the
+ second class (see A.2 above), that will connect job's signals to dialog's
+ slots
+
+ - DefaultProgress ( former KIOSimpleProgressDialog ) that will be used for
+ regular progress dialogs created by GUI Progress Server
+
+ - StatusProgress ( former KIOLittleProgressDialog ) that can be used for
+ embedding in status bar
+
+C. GUI Progress Server
+-----------------------
+ This is a special progress server.
+ - createProgress() will either create a DefaultProgress dialog or add new entry
+ in a ListProgress ( an all-jobs-in-one progress dialog )
+ - after receiving signals from the JobObserver via DCOP it will call
+ appropriate method of progress dialog ( either in DefaultProgress or ListProgress )
+ - ListProgres can be a Caitoo style dialog, kicker applet or both in one.
+
+D. Some notes
+--------------
+ 1. most of the apps will not care at all about the progress display
+ 2. user will be able to choose whether he wants to see separate progress
+ dialogs or all-in-one ListProgress dialog
+ 3. developers can create their custom progress dialogs that inherit
+ ProgressBase and do any manipulation with a dialog if they use a second
+ approach ( see A.2 above )
+
+
+Streaming
+---------
+
+ 1. We currently support a streaming "GET": e.g. file:/tmp/test.gz#gzip:/
+ works. The following should also work: file:/tmp/test.gz.gz#gzip:/#gzip:/
+ The current approach makes a TrasnferJob for gzip:/ and then adds a
+ subjob for "file:/tmp/test.gz.gz#gzip:/" which itself adds a subjob
+ for "file:/tmp/test.gz.gz".
+ 2. This doesn't extend very well to PUT, because there the order should
+ basically be the other way around, but the "input" to the job as a whole
+ should go to the "gzip:/" job, not to the "file:/tmp/test.gz.gz."
+ It would probably be easier to implement such a job in the way the
+ current "CopyJob" is done. Have a Job and make all sub-urls sub-jobs of
+ this Job.
+ 3. As a result of 1. COPY FROM an url like file:/tmp/test.gz#gzip:/ should
+ work. COPY TO does not, because that would require PUT.
+
+
+Resuming
+--------
+
+A rough note for now, just to have this somewhere :
+(PJ=put-job, GJ=get-job)
+
+PJ can't resume:
+PJ-->app: canResume(0) (emitted by dataReq)
+GJ-->app: data()
+PJ-->app: dataReq()
+app->PJ: data()
+
+PJ can resume but GJ can't resume:
+PJ-->app: canResume(xx)
+app->GJ: start job with "resume=xxx" metadata.
+GJ-->app: data()
+PJ-->app: dataReq()
+app->PJ: data()
+
+PJ can resume and GJ can resume:
+PJ-->app: canResume(xx)
+app->GJ: start job with "resume=xxx" metadata.
+GJ-->app: canResume(xx)
+GJ-->app: data()
+PJ-->app: dataReq()
+app->PJ: canResume(xx)
+app->PJ: data()
+
+So when the slave supports resume for "put" it has to check after the first
+dataRequest() whether it has got a canResume() back from the app. If it did
+it must resume. Otherwise it must start from 0.
+
+