1 Title : Guidelines for ioi competitions 2 Author : ioi scientific Committee

417 and the subsequent grading.

418

419 Each IOIn is a unique, one-time event. Almost all the

421 * software (operating system, compilers, editors, support tools,

422 libraries, grading system),

423 * officials (from Host SC members to assistants doing errands), and

424 * delegations (from all over the world)

425 will be new at IOIn. They certainly have never before operated together

426 in this mix and on such a grand scale as required by IOIn. The First

427 Competition Round has often revealed a number of deficiencies, most of

428 which are then repaired before the Second Competition Round. There is no

429 reason to believe that a first large-scale run can be made to work

430 flawlessly.

431

433 both the preliminary Regional Contests and the World Finals, there is

434 always a practice contest prior to the real contest. The practice

435 contest resembles the real contest as much as possible, except that

436 the contest problems are far fewer and simpler, and that it takes

437 less time.

438

439 It is strongly recommended that a Practice Competition is conducted at

441 phases and in that way help eliminate troubles before they can do any

442 harm. The most critical phases are the competition proper and its

443 grading. Whether translation should also be covered in the Practice

444 Competition remains to be considered.

445

447 Tasks at the Practice Competition must be feasible with minimal

448 intellectual effort. It remains desirable that, besides the Practice

449 Competition, the schedule also allows time for "free format" practicing.

450

451 There should be sufficient time between practicing and the 2nd GA

453 solutions formulated.

454

455 2ND GA MEETING (QUESTIONS): After practicing, the IOIn CC may want to

457 about the IOIn Competition. After this meeting, delegations must be

458 given the opportunity to provide final instructions to their competitors

459 before the First Competition proper (i.e. before the 3rd GA Meeting,

460 because that puts them out of touch with their competitors).

461

463 been prepared in English. The IOIn CC presents the material for the

464 First Competition to the GA. This includes the information exactly as

465 it is intended to be offered to the competitors and which needs to be

466 translated to the native languages of the competitors. It also

467 includes additional information (not to be offered to the competitors

468 until after the competition), such as background information on the

469 tasks and grading details. The GA has the opportunity to reject a task,

470 but only for compelling reasons (such as prior use). Once an approved

471 set of Competition Tasks and accompanying information has been

472 established, this material is subsequently translated by the GA. The

473 translations are published, for example, by posting the printouts on the

474 walls of the GA Meeting Room.

475

477 competitors until after the First Competition proper is over. The

478 IOIn CC should take appropriate measures to minimize opportunity for

479 any such communication.

480

481 FIRST COMPETITION PROPER: The Competition Site is prepared for the First

483 position officials, ...). Competitors enter the Competition Site (check

484 badges) and are seated at their workplaces, where they find a sealed

485 envelope with the Competition Material (including English originals and

486 a translated copy if supplied by the delegation). Additional

487 competition material is accessible through their computer. When the

488 start signal for the competition has been given, they may open the

489 envelopes and start to work, in particular, access their computer. In

490 the future, it may be possible to offer all the Competition Material in

491 an electronic form (when more advanced display technology is available).

492

493 Since IOI'94, each of the two competitions at IOIn has had THREE

495 different kind of challenge and there is some variety in the level of

496 difficulty. The First Competition is typically somewhat easier than the

497 Second Competition. The example data given in the task descriptions is

498 also available in electronic form on the computers. There possibly are

499 additional aids, such as more example data and helper programs.

500

501 How difficult a competition will turn out to be for the competitors is

503 carefully, and this is one of the toughest jobs for the organizers. If

504 the competition is too easy, then too many competitors will end up with

505 the same (high) score, thereby complicating the task of awarding medals.

506 If it is too difficult, then this has a negative impact on the image of

507 the IOI, and the effort of the organizers is wasted. The level of

508 difficulty depends on

509 * the number of tasks

510 * dependencies among tasks

511 * the difficulty of each task, which in turn depends on how hard

512 and how much work it is

513 - to comprehend the task

514 - to isolate the "real" issues

515 - to tackle these issues

516 - to design a solution

517 - to implement the design

518 - to verify the implementation

519 In view of the limited duration (five hours), it seems unreasonable to

520 include more than three challenging tasks per competition, keeping in

521 mind that context switches are a heavy burden on the competitors. With

522 fewer than three tasks per competition one cannot ensure sufficient

523 diversity.

524

525 In the first hour of the competition, written questions about the

527 any of the designated Competition Officials. Each form accommodates a

528 single question that must be answerable by Yes, No, or No Comment. The

529 competitor can pose the question in their native language. The official

530 takes the form to the Central Competition Desk where it is registered

531 (given a time stamp and tracking number), and delivered to an

532 appropriate member of the IOIn CC. The IOIn CC may approach an

533 appropriate GA member for translating a question. For this

534 purpose, GA members need to be available during the first hour of the

535 competition. The IOIn CC marks the answer on the form (the answer is

536 NOT translated). The form is returned to the competitor, after being

537 photocopied. Note that non-task-related issues, such as equipment

538 malfunction, can be reported directly and at any time (not only in the

539 first hour) to any competition official. The end of the first hour is

540 announced.

541

543 explicitly approved. They are not allowed to bring such things as

544 books, calculators, or communication equipment. Competitors are not

545 allowed to communicate with each other in any way, and they may not

546 cooperate or interfere with the work of others. Competitors

547 can get printouts of their work, and there is a mechanism for making

548 backups (e.g. by providing removable disks). During the competition,

549 refreshments are offered and restroom facilities are available.

550

551 Near the end of the competition proper, the time is announced,

553 competitors submit their work in electronic form, usually by leaving it

554 on their computer in a designated location. The competitors do not

555 take a copy of their submitted work when leaving the Competition Site.

556 The delegations will receive copies of the submitted work later.

557

559 the competitors is graded. If grading requires competitor involvement,

560 then there will first be a one-hour break. The work submitted by the

561 competitors is evaluated according to the prepared grading process, and

562 points are awarded for each task. These Guidelines strongly recommend

563 that for programming tasks, the competitors submit SOURCE CODE, instead

564 of executables. Here are some reasons to do so:

565 * Scientifically, the quality of software is judged from its design

566 and its source code, and not (solely) from the executable code.

567 * Source code is more platform independent. This makes it easier

568 to prepare competitors for an IOI Competition and to evolve the IOI

569 Competition's computing platform.

570 * Source code provides more grading options, including the

571 enforcement of more varied resource restrictions.

572 * The ACM ICPC and various Internet programming competitions

573 are successfully based on source code submission.

574

575 Traditionally, the grading of programming tasks is based solely on

577 human inspection of the program's source code. At IOI'99, a petition

578 supported by many delegations pointed out that the past grading practice

579 is too often not able to recognize and reward scientifically sound work.

580 The petition also requested that the grading would be reconsidered and,

581 if at all possible, would be made more "humane" and "realistic". The

582 spirit of IOI grading is that good work is given proper credit (as

583 opposed to the spirit of ICPC grading, where every mistake, however

584 small, is punished severely). This is a difficult issue. The least

585 that can be done is designing Competition Tasks such that partial credit

586 can be awarded reflecting partial accomplishment of the task. In the

587 future, it may be necessary to introduce other forms of source code

588 inspection besides compiling and executing.

589

591 (one per competitor). The delegations also receive all relevant

592 material (see checklist in Appendix) to help verify the results. A

593 delegation can submit motivated objections to the results on a Grading

594 Appeal Form at the Central Competition Desk. These appeals are handled

595 by the IOIn CC, which prepares responses in a closed meeting and

596 distributes them to the delegations (also see 4th GA Meeting).

597

599 the complete preliminary results of the First Competition Round are made

600 available to all GA members. Though this is not yet common practice at

601 the IOI, it is worth considering, because it helps locate grading issues

602 quickly.

603

605 all the submitted grading appeals in this GA Meeting. If the need

606 arises, the GA can vote on decisions and actions to be taken. In this

607 GA Meeting the final results of the First Competition should be

608 established.

609

611 Meeting.

612

613 SECOND COMPETITION PROPER: Identical in structure to First Competition

615

617 Competition.

618

619 6TH GA MEETING (POST-GRADING 2): Identical in structure to 4th GA Meeting.

621 7TH GA MEETING (AWARD DECISION): Once the results of both Competition

622 Rounds have been approved, the medal distribution needs to be established.

623 Although the medal distribution is almost completely determined by

624 the final scores of all the competitors, it is an IOI tradition that

625 the IOIn CC makes a proposal and the GA has the final say.

626

627 It is tempting to merge the 6th and 7th GA Meeting. These Guidelines do

629 addressed at the GA Meetings, and the primary purpose of the 6th and 7th

630 GA Meeting are best kept separated.

631

633 by at least one day without competition (see IOI Regulations).

634

635 At IOIn there needs to be ample opportunity for the GA to deal with

637 issues need attention:

638 * the format of the competition (how many competition days, how many

639 tasks per day, ...)

640 * the types of competition tasks

641 * the computing platform (operating system, programming languages, ...)

642 * the grading process

643 * the medal distribution process

644 These issues could be addressed at the 1st, 4th, 6th, and 7th GA meeting.

645

647 (S4.8) rule that the GA can "take decisions if the Regulations do not

648 anticipate a situation". If these Guidelines do not anticipate a

649 situation, the Contest Manager must make a proposal to the GA for

650 resolving the issue.

651

653 the IOI Regulations, these Guidelines define (at least) seven such

654 meetings. Furthermore, a Practice Competition is strongly recommended

655 (possibly to be made mandatory in a later version), and grading of

656 programs is based on source code submission (instead of executables).

657

659 ------- ----

660 The IOIn Competition Committee (CC) must first PLAN its work. For that

661 purpose, the COMPETITION MANAGER (CM) must produce a PROJECT PLAN. This

662 plan must

663

665 (document, form, program, file, etc.) to be delivered. Such items

666 can be composite (e.g. the bundle of all Competition Tasks); the

667 parts of composite deliverables must also be described, but this

668 may be done elsewhere.

669

671 organization of the IOIn Competition, with their name, role(s),

672 contact information, and to whom they report

673

675 (e.g. development of a special timer device)

676

677 * define a SCHEDULE, giving for each (class of) deliverable:

679 - the target audience,

680 - the starting date,

681 - the date(s) and method for review, and

682 - the date and method for final delivery to the target audience

683

685 the project and indicate how to prevent (reduce probability of) and

686 correct (reduce impact of) the occurrence of these risks

687

689 months prior to IOIn. The IOI SC reviews the plan in a timely manner

690 and provides feedback.

691

693 the IOI IC and to the Host Team executive board.

694

695 The Project Plan may EVOLVE over time, e.g. by adding new parts, by

697 that were not known earlier. However, once approved, any changes to

698 the Project Plan require separate approval by the IOI SC. This is then

699 brought up in a Progress Report (see below).

700

701 The IOIn Competition Committee must EXECUTE the approved plan.

703 The IOI Scientific Committee MONITORS the execution of the plan.

704

705 Progress Reports

707 As part of the monitoring process, the CM presents a monthly PROGRESS

708 REPORT to the Chair of the IOI SC. Each Progress Report must include

709 the following items:

710

711 * Status of deliverables (e.g. not yet started, p% completed, ready

713 changes that occurred in the reporting period

714 * Forecast of status changes for the next reporting period

715

717

719 deviates from the scheduled status according to the Project Plan

720 * Risks that have materialized during the reporting period, and

721 the corrective action taken

722 * Risks that are expected to materialize in the next reporting period,

723 and the preventive action taken

724 * Changes in the Project Plan

725

727 of whether the Project Plan has already been approved by that time.

728 That is, the first Progress Report is due eight months prior to IOIn.

729

731 ---------- ------

732 Within one month after IOIn, the Chair of the IOI SC must write an

733 EVALUATION REPORT. This Evaluation Report summarizes the IOIn

734 Competition (preparation AND execution). It is intended to help

735 fine-tune and improve the organization process. Material from the

736 Project Plan, the Progress Reports, and other deliverables, as well as

737 material produced at IOIn (the competitors' work, grading appeals, etc.)

738 will be used to produce the Evaluation Report. The Competition Manager

739 may be requested to provide additional material for the Evaluation

740 Report.

741

743 Report while still at the IOIn site and discusses it in person with the

744 CM or a knowledgeable representative.

745

747 for feedback. Upon approval it is delivered to the IOI SC and Host

748 Team executive board, and subsequently published at the IOI

749 Secretariat.

750

751 The Evaluation Report may include recommendations for changing

753 GUIDELINES STILL NEED TO BE ESTABLISHED.)

754

755 Requirements on the content of the Evaluation Report may need to be

757 use of programming languages, number and kind of clarification requests,

758 results per task, etc.

759

761 -----

762 Important roles to be fulfilled in organizing the IOIn Competition are:

763

765 * Chair of Competition Task Creators and Reviewers

766 * Competition Task Creators

767 * Competition Task Reviewers

768 * Chair of Technical Committee (dealing with hardware, operating system,

769 compilers)

770 * Chair of Grading Committee

771 * Chair of Competition Accommodations

772 * Assistants

773

775 organization, it must be clear what role(s) this person fulfills. Each

776 role reports to exactly one other role as follows:

777

779 | | | |

780 +-----------------+ | | +-----------------+

781 | +-----+ +-----+ |

782 | | | |

783 Chair of Chair of Chair of Chair of

784 Competition Grading Technical Competition

785 Tasks Committee Committee Committee Accommodations

786 || || || || ||

787 || || || || ||

788 || || Assistants Assistants Assistants

789 Creators Reviewers

790 ||

791 ||

793

795 (mostly on organizational issues) and to the Chair of the IOI SC

796 (mostly on technical issues).

797

799 are expected to consist mostly of persons from the IOIn Host Team,

800 familiar with the local situation of the IOIn venue.

801

803 ------------

804 The following deliverables are mandatory. Mentioned above are:

805

807 * Monthly Progress Reports

808 * Evaluation Report

809

811

813 * Competition Tasks, incl. practice and reserve

814 * Grading System (procedure and software)

815 * Final Results

816 * List of Delegations (with contact information)

817 * Various templates and forms:

818 - Competition task description template (for task authors)

819 - Competition task comment form (for use during GA meeting)

820 - Translation Submission Form (for use during translation)

821 - Clarification Request Form (for use during competition)

822 - Grading Appeal Form (for use after grading)

823 - IOIn Evaluation Form (for feedback at the end of IOIn)

824 * Various scripts and agendas

825

827 for delivering the content.