Home Schedule Reading Projects People

Project 5B: Scheduler - QUFiX

I am planning on making a few more minor refinements, but only clarifications and hints.

Assignment Day November 09, 2016 (Wednesday - midnight)
Due Date November 29, 2016 (Tuesday before class)

       

Collaboration Policy - Read Carefully

You can work on this project with a partner, and you are encouraged to discuss the assignment with other students or teams on the email list. You may consult (but not copy) any outside resources you including books, papers, web sites and people (but no penguins or sea urchins).

Note if you work on a team you are required to include a statement to specify the contributions of each team member. The statement can be up to several sentences long, describing the tasks of each individual [see examples here of contribution statements here - note this is in the context of a research article but it still a real world example (starting at 2nd paragraph) here ].

If you work on your own - you will get a 5% bonus provided that your base grade is above 60%). As this will require more effort.

If you use resources other than the class materials, indicate what you used along with your answer.

You have probably guessed it by now - every project will have a similar paragraph to the one above but with slight variations, of-course.

Lessons that you may have learned (or not) 1) Participate on the google MINIX group, this is a fantastic resource, 2) Use google - you won't get penalized - it is a win-win tool 3) Use the class list, but to get a reply - be courteuous and outline what resources you have tried before posting, e.g., I tried searching on google for "add a server to MINIX", I posted on the google MINIX but I did not get a reply 4) MISC - IF you do work on a team - and get stuck - try to use your partner's set-up. 5) backup frequently - your kernel will crash - and it will crash often.

Objective:

The main objectives for this project are:

The main goal for this project is to modify the MINIX 3 scheduler to be more flexible. To achieve this goal you must implement another system call and the scheduler in MINIX

 

Tutorial / References

MINIX resources programming/tutorials:

There are several MINIX resources that I found useful.

http://en.wikipedia.org/wiki/MINIX_3 (see great links at bottom).

http://www.minix3.org/doc/A-312.html (outlines the process nicely)

http://wiki.minix3.org/en/UsersGuide/DoingInstallation (newer and above, but the first listed is still useful).

http://wiki.minix3.org/en/UsersGuide/RunningMinixOnVmware(insights on running MINIX on WMware.

http://www.minix3.org/manpages/index.html (Minix man pages)

How to implement a system call:

http://www.phien.org/ucdavis/ta/ecs150-f03/syscall.html http://www.cis.syr.edu/~wedu/seed/Documentation/Minix3/How_to_add_system_call.pdfhttp://fixunix.com/minix/255067-add-system-call.html

VMWare resources programming/tutorials:

http://www.vmware.com/products/player/ (free VMware player)

http://www.vmware.com/products/fusion/ (WMWare for mac and it links to 30-day trial - we are hoping to get/setup a 12 months license for our are waiting for an approval for this - it looks like a possibility right now - please email me if you are interested in this). If we can't get it, please use VirtualBox - it is a bit slow but it works.
If you find other resources please post links on the mailing list for the class.

Coding Strategy:

And it doesn't hurt to mention this again:

As previously mentioned this project will teach you how to experiment with a real operating system kernel(s)- and as a consequence you will learn to work in such a way that deals with a computer crashes (well not quite since we are working in a virtual machine) and / or end up with an OS that doesn't work.

We suggest that you manage multiple kernels (you must have multiple kernal's anyway during the demo), and always have access to at least one working kernel. Back up your work frequently and work in small incremental steps. Incremental steps is very important. A step could be as small as one line of code, or changing the value of an assignement.

Basics:

This center piece of this project is to hack the modify the scheduling policy of MINIX. There are three tasks in the project and TWO of the taks can be completed independently. The three tasks are:

  1. Implement: a system call in MINIX (you did this already in the first MINXI project) so now you get to do it again).
  2. Implement: 2 simple scheduling policies: Lottery Scheduling and Stride Scheduling.
  3. Experiment & Evaluate: Compare and analyze the performance of the original and new scheudulin policies.
  4. Challenge: Make a dynamic scheduling policy so that a system call changes the scheduling policy on the fly (note this can be tricky because of the multiple queueus in the original MINIX). HINT: You need to flush the queues, to get all the events (jobs) in the queue.

Part 1: Scheduling:

Unless explicitly specified, all source code can be found in /usr/src/.

Currently the Minix scheduler maintains 16 queues of processes. System processes use queues 0 to 6. Queue 15 is for the idle process which only runs if there are no other processes to run. The scheduling algorithm finds the highest priority queue that is not empty and picks the process at the head of the queue. Scheduling is round robin for each queue.

In this assignment you will implement other scheduling algorithms and compare them with the current Minix scheduling algorithm, namele Lottery Scheduling and Stride Scheduling and your own custom scheduling algorithm (be creative, or non-creative - but still make a great scheduler - byt name you must name your new scheduling algorithm).

  1. Lottery Scheduling:

    The first approach to scheduling is to use lottery scheduling, as described in the Wednesday's presentation (Jesse's). System processes as described by Babak (queues 0–14) are run using their original algorithm, and queue 15 still contains the idle process. However, queue 16 contains all of the runnable user processes, each of which has some number of tickets. The default number of tickets for a new process is 5. However, processes can add or subtract tickets by calling setpriority(ntickets), which will increase the number of tickets by ntickets (note that a negative argument will take tickets away). A process cannot accumulate more than 100 tickets.

    Each time the scheduler is called, it should randomly select a ticket (by number) and run the process holding that ticket. Clearly, the random number must be between 0 and nTickets-1, where nTickets is the sum of all the tickets belonging to processes in the ready queue (processes that are blocked are ineligible to run). You may use the random() call (you may need to use the random number code in /usr/src/lib/other/random.c) to generate random numbers and the srandom() call to initialize the random number generator. A good initialization function to use would be the current date and time.

    New processes are created and initialized in kernel/system/do_fork.c. This is probably the best place to initialize any data structures.

  2. Stride Scheduling:

    This part will be challenging because now you are the scientist: you just read about a cool new sheduling algorihtm called stride scheduling here:

    http://waldspurger.org/carl/papers/phd-mit-slides.pdf (slides)
    http://www.waldspurger.org/carl/papers/stride-mit-tm528.pdf (paper)

    and now you want to implement it in MINIX. You already implemented lottery scheduling and now you have you "basically" have to implement a deterministic version of lottery schedulign using the guidance of the slides and paper above. In your design document note what you did and what decisions you made on how to implement stride scheduing in MINIX.
  1. Challenge/Bonus: Make a better scheduling and the above algorithms, be creative. Make sure you can demonstrate that your scheduler performs better under certain workloads.

     

Please read the background section in 2009 assigment to undertand the MINIX coding structure better. Please note that there are some differences between 3.1.7+ and previous versions of MINIX, these are however no major differences in the context of this assignment (and as confirmed by Babak who has already coded last year's assignment in 3.1.8).

http://www.cs.uga.edu/~maria/classes/4730-Fall-2009/project4minix-scheduler.html

Part 2: Experiment & Evaluate:

Compare the new schedulers with the original scheduler. This will be done by running a workload generated by two programs: scanfiles.c and timewaste.c. The program in scanfiles.c is an I/O bound program while timewaste.c is a CPU bound program. Both of these are found on the assignments will be available shortly. Your comparison should be based on the time it takes to complete scanfiles.c. You can use the time command which will give you the real time used by a process. You should average times observed over a series of runs since times will fluctuate (we suggest 3 to 5). There is no need to reboot MINIX or log in again. You should run the trails in quick succession. You should include the stastics and your conclusion in your report with your results and explain why you think you got the results you have.

Here is scanfiles.c and here is timewaste.c

Part 3: Scheduling System Call:

Challenge: You need to create a system call so that it changes the scheduling policy on the fly (note this can be tricky because of the multiple queueus in the original MINIX). Hint: You will need to flush the queues.

 

Hints

Last year - students requested do know about interrupt handling in order to understand the life cycle of a process better. I have included the discussion below.

MINIX Interrupt Handling (optional reading to understand the life cycle of a process)

If you are interested in the life cycle of a process and details of the mechanics switching process -- read on. As discussed in class an interrupt handler is responsible for the grunt work of switching and 'restarting' (runnable) ready processes (or removing a preempted process form the CPU). The MINIX code for this (that you should not need need to modify) is in mpx386.s (in src/kernel/) (and is in assembly code). Here is the flow of restarting a process using the interrupt handler. Hope that this will give you a better picture.

  1. CPU is stopped.
  2. The CPU automatically goes into kernel mode with further interrupts disabled.
  3. Values are pushed to registers (e.g., EIP) onto the stack
  4. Hardware detects nature of interrupt via a number called the IRQ
  5. CPU uses IRQ number to index into the interrupt descriptor table
  6. The interrupt is handled accordingly (via IRQ number).
    Here the instruction pointer is automatically set to start the execution of the appropriate interrupt handler code in MINIX, this code is in the assembly code file mpx386.s (src/kernel) and the entry point, for clock interrupt handling for instance is IRQ - 0 (I believe - please double check this detail) at _hwint00: hwint_master(IRQ number) in mpx386.s
  7. _restart in mpx386 is the code that runs the process at
    _next_ptr (or _proc_ptr) that is set in
    pick_proc (kernel/proc.c).
    proc_ptr at end of restart will point to the slot of the process table for the process that runs next (which might be the same process that was interrupted).

We assume that you are already familiar with makefiles and debugging techniques from your earlier project in this class or from courses such as CS 1730 or the UNIX tutorials held the first week of class by the department. If not (which is unlikely), this will be a considerably more difficult project because you will have to learn to use these tools as well. Alternatively Babak can help you get set up - but now you need to start EARLY.

This project doesn't require a lot of coding (typically fewer than 250 lines of code), but it does require that you understand how to use MINIX and how to use basic system calls. You're encouraged to go to the class discussion section or talk with the course staff during office hours to get help if you need it.

You should do your design first, before writing your code. To do this, experiment by inserting debugging print statements if it'll help. It may be more fun to just start coding without a design, but it'll also result in spending more time than you need to on the project.

IMPORTANT: As with all of the projects this semester, the key to success is starting early. You can always take a break if you finish early, but it's impossible to complete a 20 hour project in the remaining 12 hours before it's due....

Project groups

You may do this project with a project partner of your choice. However, you can't switch partners after this assignment, so please choose wisely. If you choose to work with a partner (and we encourage it), you both receive the same grade for the project. Note if you work on a team you are required to include a statement in a single file called partner.txt to specify the contributions of each team member. The statement can be up to several sentences long, describing the tasks of each individual [see examples here of contribution statements here - note this is in the context of a research article but it still a real world example (starting at the 2nd paragraph here) ]. Be sure to include the names (first and last) of both partners in the file.

Submitting:

Submitting on odin via the submit command.

  1. You must include a design document for this project (more on this shortly).
  2. Create a file named mypatch that contains the patch to the Minix kernel to print the requested debug information (as described above). Please note that this patch must cleanly apply to a fresh source tree. You may submit multiple patches (i.e., at a minimum you may only need to submit patches for the files you changed)
  3. Your files must be in a directory named project4.
  4. Put all the materials needed discussed above in the above directory (including your README.txt file)
  5. Submit via the 'submit' command (while on odin.cs.uga.edu)
  6. You will also need to turn in 'talking points' for the demo - this documents is not be due at this time but it will be due before the demo.
  7. Example submit command:

    {odin:maria} submit project5B csx730

You will need to demonstrate your projects during class time (everyone will be in the same room). For this demo you need to prepare a short (talking) script 'convincing' us that you have implemented the project appropriately and correctly.

 

Design Document:

Your design document should describe the design of your assignment in enough detail that a knowledgeable programmer could duplicate your work. This includes descriptions of the data structures you use, all non-trivial algorithms and formulas, and a description of each function including its purpose, inputs, outputs, and assumptions it makes about the inputs or outputs. A sample design document is available here.

Grading Criteria:

Coming soon! (look at semaphore and old scheudling assignment from 2009 to get an idea how projects with a design documents are graded). We will use a similar grading strategy.

Projects will not be graded without a live demo.