memory-hotplug: fix pages missed by race rather than failing

If race between allocation and isolation in memory-hotplug offline
happens, some pages could be in MIGRATE_MOVABLE of free_list although the
pageblock's migratetype of the page is MIGRATE_ISOLATE.

The race could be detected by get_freepage_migratetype in
__test_page_isolated_in_pageblock.  If it is detected, now EBUSY gets
bubbled all the way up and the hotplug operations fails.

But better idea is instead of returning and failing memory-hotremove, move
the free page to the correct list at the time it is detected.  It could
enhance memory-hotremove operation success ratio although the race is
really rare.

Suggested by Mel Gorman.

Signed-off-by: Minchan Kim <minchan@kernel.org>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Acked-by: Mel Gorman <mgorman@suse.de>
Cc: Xishi Qiu <qiuxishi@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h
index 105077aa7685c61dbb66cabc8473302283a0e3a5..fca8c0a5c1883d6079d39d5e11a7ca0b9fc4bc39 100644 (file)
--- a/include/linux/page-isolation.h
+++ b/include/linux/page-isolation.h
@@ -6,6 +6,10 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count);
 void set_pageblock_migratetype(struct page *page, int migratetype);
 int move_freepages_block(struct zone *zone, struct page *page,
                                int migratetype);
+int move_freepages(struct zone *zone,
+                         struct page *start_page, struct page *end_page,
+                         int migratetype);
+
 /*
  * Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE.
  * If specified range includes migrate types other than MOVABLE or CMA,

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0e94f6485fd53e50d5db344cc154c1fbd1d7b5c5..38b2e3862ca208c746437bc984d3d01328eb928f 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -924,7 +924,7 @@ static int fallbacks[MIGRATE_TYPES][4] = {
  * Note that start_page and end_pages are not aligned on a pageblock
  * boundary. If alignment is required, use move_freepages_block()
  */
-static int move_freepages(struct zone *zone,
+int move_freepages(struct zone *zone,
                          struct page *start_page, struct page *end_page,
                          int migratetype)
 {

diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 6744235d2d0e18d984e94c824de36db46b74da0b..c7445d1e1ccf1821049756f86839b15dcc95ee2f 100644 (file)
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -201,8 +201,19 @@ __test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn)
                }
                page = pfn_to_page(pfn);
                if (PageBuddy(page)) {
-                       if (get_freepage_migratetype(page) != MIGRATE_ISOLATE)
-                               break;
+                       /*
+                        * If race between isolatation and allocation happens,
+                        * some free pages could be in MIGRATE_MOVABLE list
+                        * although pageblock's migratation type of the page
+                        * is MIGRATE_ISOLATE. Catch it and move the page into
+                        * MIGRATE_ISOLATE list.
+                        */
+                       if (get_freepage_migratetype(page) != MIGRATE_ISOLATE) {
+                               struct page *end_page = page +
+                                               (1 << page_order(page)) - 1;
+                               move_freepages(page_zone(page), page, end_page,
+                                               MIGRATE_ISOLATE);
+                       }
                        pfn += 1 << page_order(page);
                }
                else if (page_count(page) == 0 &&