Turning this into a separate issue, to make tracking a bit easier - it's based on this: https://virtualmin.com/node/22867#comment-126823
Agreed, a custom wizard framework would be a good idea. Oddly I haven't used wizards much in Webmin/Virtualmin previously, which is likely why this hasn't been done before.
As mentioned here, it seems that the existing "custom commands" feature in webmin would provide for a solid foundation to design a dynamic wizard framework on top of it - where a wizard would be an array of wizard pages, and each page could contain fields (analogous to the ones used by custom commands) or custom commands directly.
Supporting this, would make it possible for people to easily create wizards using custom commands. Based on playing with the code, it seems there's very little missing to make this happen actually.
So here's a random collection of thoughts on integrating wizards and custom commands:
- being able to package wizards/custom commands analogous to wbt packaging would be useful to easily exchange them, i.e. web-based downloading/installation
- some additional field types would be useful, especially pre-populated lists - for things like local vhosts, configured cluster servers etc
- currently, the target server must be specified when editing the command, it may be useful to provide an option to select this at runtime, i.e. provide a new field type "webmin servers" and an option where the server can be selected when running the command
- the notion of having prerequisites (analogous to webmin modules) should probably also be supported by custom commands, i.e. to check if certain packages are installed - sort of like "precondition"
- a wizard will typically involve multiple pages, but also multiple stages, where commands may need to be running in the background, i.e. scheduled via cron for example
- it would be very useful to directly expose the webmin/virtualmin CLI tools to the custom commands feature
Ideally, the UI part would be kept separate, so that wizards could also be run via the CLI, i.e. using GNU readline() then.
Comments
Submitted by JamieCameron on Thu, 01/09/2014 - 16:11 Comment #1
I suppose this could be done.
Although once you reach this level of complexity, why not just write a Webmin module? Given a little knowledge of Perl, it isn't that hard.
Submitted by wocul on Thu, 01/09/2014 - 18:16 Comment #2
Although once you reach this level of complexity, why not just write a Webmin module? Given a little knowledge of Perl, it isn't that hard.
My Perl knowledge really is very basic, while I could probably get up to scratch quickly, I feel that it would be a good thing to provide wrappers on top of "custom commands" to create simple wizards, e.g.:
Once I write a module, it will be a module for a single purpose only - it will be of little use to other modules - as mentioned elsewhere, there's a certain disparity between features available via virtualmin CLI and via the web frontend or RPC.
So, the idea is to provide a common framework that handles these detaiils, so that people can focus on coming up with wizards + commands to implement them, while the framework would ensure that these wizards would satisfy the most common webmin use-cases. (web/cgi, cli, rpc).
I agree that creating a simple module may not be difficult, but many of them cannot be easily leveraged in other places - also, there's the whole theme handling challenge - by wrapping custom commands and presenting them to the user via wizards, usability would be greatly improved.
Generalizing the wizard-code in webmin wouldn't be very difficult - but very soon, people would require custom fields, validation and support to run commands - that's what the "custom commands" module is all about - so it would be better to identify the hooks required to reuse the "custom commands" module for such purposes, it is already extremely powerful obviously - and would probably only require a handful of augmentations to make it usable as the backend for "wizards".
Just look at the amount of custom code there is doing basically the same things: - presenting steps in a wizard-like fashion - running commands - processing output
I supposed that, had "wizards + custom commands" been available years ago, that many existing modules would have been implemented that way - because it is so much more straightforward to do, while also reducing maintenance overhead significantly.
Also, this would be a great opportunity to generalize and unify things - for example, based on following the "theming" discussions, that's one messy aspect in webmin - arguably, this is not made easier by having many modules using the existing API.
Now, by extending the "custom" module in webmin, wizards could be created using "custom commands" as their back-end - which in turn means that people could create dozens or even hundreds of such "wizards", while there would still only be a single place to update, when it comes to supporting other themes.
Then, there's a number of features that may often only be available as either a CGI, CLI or RPC feature - a wizard framework could automatically encourage a loosely coupled design where CLI code is required, and used by the RPC/CGI modes - so that a wizard would not be complete without filling in all the required placeholders.
Basically, I am thinking in terms of a handful of features, such as supporting recently discussed tools like:
Now, I may be able to create a dedicated Perl module for one of these, and it would probably take me several weeks - but looking at your powerful "custom commands" feature, I can't help but thinking that if this could be extended, and become usable by a simple "wizard" framework, most of these could be tackled in just a few hours.
Thus, it's also a matter of productivity at some point - I could spend 30 hours coding a dedicated module for a single purpose, or I could spend 20 hours implementing a handful of "wizards" directly.
it's obvious that the "custom" module is so much more straightforward than creating Webmin modules from scratch using Perl - the latter requires familiarity with lots of APIs in Webmin. The former is all about grouping "custom commands" together with some pre-defined webmin/virtualmin hooks.
For example, let's talk about the idea to support multi-stage migrations, where TTL setting and checking (via dig) is part of the game - it would probably take me at least 15 hours to come up with a module for that - but it only took me 15 minutes to come up with a handful of custom commands doing each step.
Now, the only thing missing is an ability to use these commands as elements in a wizard, and feed output from one stage to another stage (command).
Don't undererstimate the improved user experience and usability improvement here - writing Perl code is obviously straightforward to you guys, but writing non-trivial Webmin modules from scratch is a completely different thing - it isn't easily done without spending many hours reading up on various APIs.
Here's another example: remember the "supporting remote vhost migrations over SFTP" discussion we had recently ?
In building blocks, this just entails having: - a way to modify DB credentials - a list of vhosts - destination details (server, port, user, password) - transferring the files of the vhost - patching it up to use the new environment (DB & paths)
Conceptually, I know exactly how to implement each step as a "custom command" within 5-10 minutes - but doing this directly in Perl/Webmin would be a different thing obviously.
Basically, it is about empowering your users to create such extensions themselves, rather than just filing feature requests and having to wait several months for a new feature.
Submitted by JamieCameron on Thu, 01/09/2014 - 20:14 Comment #3
The Webmin API itself does provide a framework for creating these kinds of modules, so you wouldn't have to implement the whole thing from scratch.
Multi-stage Virtualmin migrations is one that we definitely will implement. But a really generic "app builder" framework is likely to be almost as complex to use as writing a module in Perl.
Submitted by wocul on Fri, 01/10/2014 - 02:42 Comment #4
But a really generic "app builder" framework is likely to be almost as complex to use as writing a module in Perl
Exactly, you are proving my point by saying "writing a module" - supporting all use-cases may be complex and difficult, and it may be as tricky as writing a module from scratch - but the point being is that this only needs to be done once, and can be used later on, by people who may be more familiar with shell scripting than Webmin/Perl scripting.
So, let's try to walk before we run:
You already said that you like the idea of generalizing the existing wizard code to allow it to be reused, right ?
If that's the case, wizards (pages) will inevitably need some way to specify fields for each page - that's clearly overlapping with the support for parameter fields already found in the "custom commands" module.
The other thing is, wizards also need a way to run "commands" at some point.
So it makes absolutely sense to model wizards on top of custom commands, even if not each page should actually "run" a command, but only gather information required by other pages.
This alone would already be very useful, and could also help clean up some things in Webmin.
However, if you think that this is beyond the scope of Webmin or not worth implementing, just let me know, so that I can look into it myself. I'd just ask you to tell me if there's anything that you are likely to work on, to avoid duplicate work, specifically in terms of custom commands and wizards.
EDIT: If I am not mistaken, I could use webmin's "clone module" feature to start off with "custom commands" and try to merge in the stuff available in "wizard-lib/wizard.cgi", right ?
Also, maybe the "custom command" docs should mention this: http://sourceforge.net/p/webadmin/discussion/600155/thread/254bc40e#cb40 It took me quite a while to figure out, because it isn' t mentioned anywhere apparently: http://doxfer.webmin.com/Webmin/CustomCommands
Submitted by JamieCameron on Fri, 01/10/2014 - 13:12 Comment #5
By "generalizing wizards", I meant adding functions to the Webmin perl API that allows making wizards easier. It wouldn't be usable directly from the custom commands module - only from new module code.
Submitted by wocul on Fri, 01/10/2014 - 23:59 Comment #6
sure, that sounds like a very good first step - and it would be a precondition for using the same code in the "custom commands" module. I'd just to suggest to keep our options open - having a module that creates wizards using such an API should be fairly straightforward. Indeed, one could then even use the same framework to implement script installers. After all, it's just a matter of identifying common building blocks used in most script installers, and turning them into custom commands that can be used by wizards for installing scripts.
Now, let's turn this around a little, I realize that you guys are not overly fond of following this route, so I am wondering how would you suggest to proceed if others want to look into this instead ?
Thank you
Submitted by JamieCameron on Sun, 01/12/2014 - 00:17 Comment #7
I'd start by looking at the existing UI API functions in Webmin's ui-lib.pl file, and seeing how hard it would be to add a Perl function for wizard creation.
Submitted by wocul on Sun, 01/12/2014 - 11:31 Comment #8
Thank you for the pointers, you mentioned that the virtualmin git repo would be private, and that I should directly look into /usr/libexec/webmin - so I turned that into a new git repo, but it seems that there's quite a bit of "live" data written to that directory apparently ? I mean all the "history" stuff in various sub folders - now, it seems that config files are written to /etc/webmin - is there any way to also write "history" stuff to some other location so that I can keep the webmin directory clean (i.e. in git) or what would you recommend ?
Thanks